What is Surroga?

Surroga is a keyboard & mouse automation utility for macOS. It can automate every thing you can do with your Mac, even iPhone.

Surroga can be downloaded on AppStore. We continue to make Surroga the best automation utility ever to fit you needs. We need you help to shape the future direction of new features and improvements. If you have any comments about Surroga, please feel free to let us know. We will reply every feedback email.

Content

What is Surroga?
Script
Statement
Control Flow
- If, Then, Else
- Loop
- Continue
- Call
- Return
- Exit
Condition
Actions
Variables
Examples
Performance
Expert Mode
In-App Purchases
Privacy & Security
Privacy Statement

What is Surroga?

Surroga is a keyboard & mouse automation utility for macOS.

Send keystrokes or typing.
Move, click, and drag anything via mouse.
Calculate numbers.
Text recognize & process.
Image recognize & click.
Flexible screenshots.
Flexible copy & paste.
Conditional delay.
Open 3rd-part Apps.
Call terminal commands.
Nested scripts and loops.
Variables
Conditions
Programmable UI, include sound, flash-screen and pop-up.

Tips: The app will need ask for "Accessibility" permission at the "Privacy & Security" at the System Settings.

Script

Record Script

Surrga can record user operations in the specified window.

Open the left sidebar, click (Record), then choice the application to record.
Do mouse operations in the window of the selected application.
Click (Stop), or press Shift-Command-R, or press Shift-Command-ESC, to stop recording.
Enter the name of the new script. then click Save.

The new recorded script will appear in the script list at the left sidebar.

Remove Script

To remove a script, select it via the script list at left sidebar, then press the DELETE key on the keyboard. Or, click the script via the mouse right button, the click the Delete option in the context menu.

Export & Share

Select a script, then click , choice Share or Share as Protected.

The protected script is not editable, and will not create the logs.

Statement

Statement group expressions and control the flow of execution.

Sort Statements

Drag the statement via mouse to change its order in the statement queue.

Delete a Statement

In the script editor, click a statement via the mouse right button, the click the Delete option in the context menu.

Tips: Don't click in the textfield. Or else, the context menu is not include the Delete item.

Control Flow

If, Then, Else

'If', 'Then', and 'Else' are collectively referred to as 'conditional statements'.

Conditional Statements

By default, one statement include only one action. When a statement is executed, this action will be executed.

On the other side, we can insert an IF sub-statement into a statement. The IF sub-statement is consist of one or more Conditions . The action of the statement, so called THEN sub-statement, will be executed only when conditions are true. If Conditions are false, this statement can be skiped, or execute the action in the ELSE sub-statement.

AND, OR

There are two combination types that be used between multiple conditions in the same statement.

AND: The actions contained in a statement will only be executed when all conditions of that statement are met.
OR: When any one of the conditions in a statement is met, the actions contained in that statement will be executed.

When a statement contains multiple conditions, we can setup their combination type:

Condition Combination

Tips: Only one combination type can be used between multiple conditions in the same statement. In other words, all conditions contained in a statement must either be AND conditions or OR conditions.

Loop

Surroga doesn't support LOOP statement yet. Instead, it support flexible repeating at script level by Repeat-limited, Repeat-infinite, Repeat-until or Continue statement.

To initialize a loop, consider two scripts, main and sub-script. In the main script, we initialize the variable name VAR by assign a value(1) to it before call the sub-script. In the sub-script, it increase the VAR by 1 each time, and repeat until VAR is greater than 5. Surroga ship with a on-board script named "Counting". To view the results of this script, open a window that support text inpurt (typing keys), the click the system status bar -> Surroga -> Run -> General -> Counting.

Repeat Until

Continuously and repeatedly execute the current script until a certain condition becomes true.

Continue

Bypass the remainng statements in the current script, move on to the beginning of the current script, re-execute the current script from the beginning.

In Surroga, Continue statement is used to implement loop at script level.

Call

Call another script. At this time, the caller is often referred to as the parent script. The called script is referred to as the child script.

NB! Surroga allows scripts to have the same name. Therefore, when calling another script, one should be careful not to mistakenly call an unrelated script with the same name, especially when a user-defined script has the same name as a built-in script of Surroga.

Return

RETURN statement will abort the current script, and return to its parent script (the caller).

RETURN statement doesn't returns any value to its parent script. Instead, if we want to do this, assign the value to one of variables before return, then read the value of the variable in the parent script. In Surroga, the scope of variables are global. Variables are shared by all scripts.

In Surroga, it is implemented as an action.

Exit

EXIT, also called ABORT, END or STOP statement. This statement only only abort the running of the current script, but also its parent script, ..., parent of parent. Finally, it abort the 'Root' script.

In Surroga, it is implemented as an action.

Condition

Window Pixel Color Is Similar

Get the pixel color at a specified position in the window and compare it with a specified color. If they are similar, it is judged to be true. Otherwise, it is judged to be false.

NB! This condition includes several hidden conditions. If the window screenshot fails, or if the pixel position is incorrect, or if they do not match, then the result is determined to be false.

NB! If you need to compare the colors of multiple pixels in a window at the same time, it is recommended to take a screenshot of the window first, and then perform 'color comparison from the image', that is, compare these pixels within the captured image content, which will be more efficient.

Color Is Similar

If both items being compared are colors and are similar, then it is judged to be true. Otherwise, it is judged to be false.

NB! This condition includes a hidden condition "both must be colors". If either one is not a color (especially when the variable content is being compared), then the result is judged to be false.

Image Is Similar

If both items being compared are images and are similar, then it is judged as true. Otherwise, it is judged as false.

NB! This condition includes a hidden condition "both must be images". If either one is not a image (especially when the variable content is being compared), then the result is judged to be false.

Image Contains Text

If the image specified by the former contains the text specified by the latter, then it is judged to be true. Otherwise, it is judged to be false.

NB! This condition includes the hidden condition "the former is an image." If the former is not an image (especially when the comparison is with variable content), the judgment result will be false.

Actions

Keystroke

Type a single keystroke, pretty much any kind of keystroke you can type on the keyboard.

How to specify the key

To specify the desired key to press, open the action editor, expand the action details, and click the key specified by the action with the mouse; that key will be highlighted. Then press a key on the hardware keyboard. The editor will record the pressed key and assign it as the key for that action.

Tips: When recording the keystroke, please avoid conflict keys that used by Surroga, other apps or macOS as the hot-key.

Hardware Assisted

For certain special scenarios or specific applications, special dedicated hardware is required to assist with keyboard operations. At this time, the Hardware Assist option needs to be enabled in the action editor (Tip: You must enable Expert Mode first to see this option).

Hardware Assisted

Typing Text

Continuously press a series of keys on the physical keyboard in the predetermined order and speed to automatically input a piece of text.

Tips: This text must be consist of keys on the hardware keyboard. No all languages are supported, e.g. Chinese.

Tips: Before take this action, please make sure that the target window has been opened/activated already, and focus the text-editing UI-control. Or else, these key-strokes will be ignored by macOS.

Hardware Assisted

For certain special scenarios or specific applications, special dedicated hardware is required to assist with keyboard operations. At this time, the Hardware Assist option needs to be enabled in the action editor (Tip: You must enable Expert Mode first to see this option).

Mouse Operations

Move the mouse cursor, click the mouse button, or dragging.

Move the mouse cursor to

text in window / on screen.
color in window.
image in window.
point at screen or in window.
relative to the current position.

Click the mouse button at the calculated position.

Click on text in window / on screen.
Click on color in window.
Click on image in window.
Click at point in window / on screen.
Click at the current mouse cursor position with offset.

Tips: Before move the mouse cursor or click the mouse button in the window, please make sure the window is already open and bring to front. See also: Open Application Action.

Similarity

In the process of searching for images and colors, Surroga will calculate the similarity between each 'suspected' object and the target object. If the similarity is greater than the specified threshold, the match is successful.

Therefore, in operations like searching for images or colors, the higher the similarity parameter, the more similar the found objects are to the target object, but the less results. If a mismatch result is found, it is recommended to raise the similarity threshold. If nothing can be found, it is suggested to lower the similarity threshold.

Tips: The real color on iPhone/iPad/Mac may be dynamic according to current system appearance (dark or light), zoom ratio of iPhone Mirroring window. So, the color parameter of the Click-Color action must fit the real environment.

Performance

To accelerate the image recognizing,

We should use the smallest size of the sub-image to search for. The suggested size is 32 X 32.
The sub-image should contains unique content other than the remains in the original image
The color of the top-left pixel of the sub-image should be the least color in all pixels of the original image.

Extra Match

When performing mouse operations on text and images, in order to handle cases where the window contains multiple target objects (texts, images), some additional matching criteria can be used to filter out irrelevant targets. For example, we may require that the target object is nearby another specified text or image.

Click Text Nearby Text

As shown in the figure above, we want to click on green 'Coordinates', but there are three instances of 'Coordinates' in the image. To eliminate interference, we can set it to only click on the 'Coordinates' that is near the 'Screen'.

Screenshot

Take a screenshot of screen or window.

We can specify the content to capture:

Entire Screen
Selected Portion of Screen
Selected Window.
Selected Portion of Selected Window

To specify an region (portion of screen or window), you need to define the position and size of that region. The origin is at the top left corner of the screen or window. When specifying a screen region, what is specified is the absolute position of that region on the screen. When specifying a window region, what is specified is the relative position of that region within the window. For a region which size is (100,100):

the top-left corner of the window: (X: 0, Y: 0, W: 100, H: 100)
the top-right corner of the window: (-100, 0, 100, 100)
the bottom-left corner of the window: (0, -100, 100, 100)
the bottom-right corner of the window: (-100, -100, 100, 100)

Tips: The action will need ask for "Screen & System Audio Recording" permission at the "Privacy & Security" at the System Settings.

Tips: When a window is in the foreground and background state, its displayed content often varies. It is recommended to open the application before taking a screenshot of the window, ensuring it is in the foreground.

Window Title

The most macOS app shows more than one windows at the same time. For example, if we click the about option of an app, this app will shows at least two windows, the main window and the about window. For Safari, every Tab own its window. We can specify the Window Title to distinguish these windows. For Safari, the window title, in most cases, is the title of Tab.

Tips: For Safari, if we specify the title of the window, in others words, the title of the Tab, this Tab MUST be the current Tab of Safari. Or lese, the script will report 'Window Not Found'."

Recognize Text

Recognize text from image or content in window.

Obviously, it's possible for an image to contain more than one piece of text. So, the results of recognizing-text is array of text, instead of just one text in a basic variable. For example, for an image that only contains a number, 32, this actions returns the array ["32"]. Then, use Get the First Element in Array action to get 32。

Tips: DON'T look the text recognition results in the variable as a basic string, specially when the image contains only one word.

The result of recognization may be different according to recognization language. For example, if we setup the recognization language to English, it can not recognize Chinese in the content.

Tips: If the action 'Recognize text' can successfully recognize text, but the condition 'Text contained in images/windows' cannot recognize the same text, it may be because the languages set in these two cases are different.

Text to Speech

Text-to-Speech (TTS) can produces synthesized speech from text in various languages, thereby supporting automated voice control, making it particularly suitable for scenarios that require voice control (such as certain iPhone applications).

TTS

Tips: If the language does not match the actual language of the text (for example, setting the language to English while the text is actually in Chinese), it may not be able to read the text correctly.

Popup Alert

Pop-up a modal dialog to show something or input something.

Popup Alert

Tips: There is only one model dialog can appear on screen at the same time. If more than one actions are executed to pop-up dialogs, only the first one will appear.

Send Notification

Send a local notification that showed on the current device. Notifications are used to show the status of running script, or notify users to do necessary operations.

Send Notification

Tips: This action just request the Mac for the notification. There is no guarantee that the notification will be showed. Please visit Apple Mac User Guide for more infomation.

Set Status Item Title

Setup or remove the title of Surroga on system status bar. This action can be used to indicate the current state of the system or the current running state of the script.

Tips: The title has a maximum length of 20 characters and should not contain special characters.

Variables

A variable is a container used to store content such as text, number, or image. Variables will make your scripts more powerful and flexible.

Tips: Please switch to Expert Mode before using variables.

Variable, Array & Dictionary

There are three kinds of variables are supported in Surroga.

Variable: Basic variables also called general-purpose variables. They are used to store text, numbers and images.
Array: Array is an ordered series of elements. We can get or set the element by its index in an array.
Dictionary: Dictionary is a collection whose elements are key-value pairs. We can use a key to retrieve the corresponding value, which can be any object.

Pasteboard

In Surroga, the pasteboard is ofen used as the special variable. Using the clipboard is more convenient than using regular variables in most cases, especially when users are developing and debugging scripts, as they can easily observe the execution status of the current step. However, for a released script, we should still avoid using the clipboard and instead use regular variables.

The access efficiency of the clipboard is far lower than that of variables. When an application accesses the clipboard, it needs to pass data back and forth between the system and the application, which incurs a significant overhead.
In addition, the clipboard is globally shared, making it easy to interfere with each other.

User-defined Variables

In addition to built-in variables, users can also define their own variables, commonly referred to as user-defined variables, custom variables or named variables.

User variables are all basic variables, accessed by name. For example,

Assign the variable named Price to value 29.9.
Get value of variable named Price.

Scope

In Surroga, the scope of variables are global. Variables are shared by all scripts.

Tips: Therefore, we must be careful to avoid interference between the scripts.

Examples

Let's discusses some common approaches to script writing.

Suppose there is an application scenario that monitors a certain area on the screen. If this area changes to the preset image A, the custom key '1' on the keyboard is pressed. If this area changes to the preset image B, the custom key '2' on the keyboard is pressed.

The above requirements can refer to the following logical flow.

Set the script to an infinite loop, which contains the following process:

Capture a specified area of the screen
If it's similar to image A, then press button 1.
If it's similar to image B, then press button 2.

Among them, image A and B can be obtained by taking screenshots.

The above script is simple enough and very suitable for beginners. However, it hides too many details. If the execution result is not as expected, it is difficult to observe which step went wrong. To address this, we can try to break down the steps and rewrite them in the following form:

Capture a specified area of the screen, output to pasteboard
Calculate the similarity between the image in the clipboard and image A, and output it to variable 1.
Calculate the similarity between the image in the clipboard and image B, and output it to variable 2.
If variable 1 is greater than the specified value, then press button 1.
If variable 2 is greater than the specified value, then press button 2.

Then, enable expert mode, open the log interface in the right sidebar, run the script, and observe the execution process.

Performance

Disable Logs

Disable Log and File Log to improve the script performance.

Optimize for Screenshot

There are some tips to optimize the performance when capture screen content.

Choice optimized for Performance instead of Quality.
Reduce the size of the target window as possible. The best way is expose the target region to capture only.
Output the captured image to Variables instead of Pasteboard.

Optimize for Recognizing

There are some tips to optimize the performance of recognizing content. For example, when recognize text, click/move/drag on text, recognize barcodes or qrcodes.

Disable Build Analysis Image option. This option is used to analyze the execution of script, but it cost heavy.

Prevent App Nap

If Surroga's window is completely hidden behind other windows for a long time, macOS will put it into a 'App Nap' state, which reduces its performance to enhance the performance of the foreground window and save energy. However, if we want Surroga to run at high performance in the background for a long time (for example, to monitor the content of a certain window in real time), we can minimize Surroga's window (instead of cover it by other windows).

Expert Mode

Surroga supports two work modes.

Beginner: Essential Features. It is recommended for beginners. It's the default mode.
Expert: All Features, with deep and precise controls, steep difficulty curve, highly challenging. It is recommended for experienced experts.

Open Surroga > Settings > Advanced, switch between Beginner and Expert.

Advanced Mode

Tips: When you can't find the feature you want, please try turning on Expert Mode.

In-App Purchases

Paid Features

Paid features include:

Variables
Calculate
Conditions
Control Flow(IF/ELSE/RETURN, etc.) & Error Handling
Export & Import
Recording as Script
Advanced Parameters for Actions

For example, when insert a new action into script, some actions in the candidate action list are disabled for picking. It means these actions are paid features. They will be unlocked after payment.

Non-Consumable vs. Subscription

Monthly-Subscription, Yearly-Subscription and Pro shared the same features but payment. Both of three unlock all features. Subscriptions are for monthly or yearly. Pro is lifetime.

Activate

Still locked after payment?

Surroga embedded a built-in shop provided by Apple. This shop is fully maintained by Apple. If the payment was completed, Apple should tell the application when network available. But, if Apple is not, there are some sugguestions below:

Restart Surroga and try again when network available.
Click the Restore button in the built-in shop in application.

Privacy & Security

Surroga can perform lots of tasks of applications and control your Mac. As such, it needs your security permissions to do these jobs for you. These permissions may vary according to different macOS versions. In most cases, the app will need ask for these permissions at the Privacy & Security at the System Settings.

Privacy & Security

As shown in the above figure, if both items in the image are displayed in green, it indicates that this application has been granted system authorization.

Accessibility

Allows control of your mouse and keyboard, and monitoring your mouse and keystrokes. It is required for following app features:

Automate a keystorke
Automate a mouse movement/clicking/dragging

Screen Recording

Allows capture the screen. It is used for following app features:

Automate to capture the user-specified window/screen.
Automate to text recognize on the user-specifed window/screen.
Automate to click the user-specified text/image on the user-specifed window/screen.
Automate to search the user-specified window/screen for the user-specified image/text/color.

Privacy Statement

Data Not Collected.

The developer does not collect any data from this app.

Which app features will use screen recording?

Automate to capture the user-specified window/screen.
Automate to text recognize on the user-specifed window/screen.
Automate to click the user-specified text/image on the user-specifed window/screen.
Automate to search the user-specified window/screen for the user-specified image/text/color.

Accessibility and Screen Recording permissions are needed to implement the essensial features of this app. The storken keys or screenshots are automated by the user's own, and never send to the developer. Please visit Privacy & Security for more infomation.

What data does your app collect via screen recording?

The data, user-specified image or text, was recorded by the user's own willing. No data was collected by this app or the developer.

For what purposes are you collecting this information? What plan for uses of this data.

No data was collected by the developer. No plan to use.

Will the data be shared with any third parties?

No.

Where will this information be stored?

This information was stored in device RAM. It will lost if app quit, or device shutdown/reboot. If the users choice output the data to system pasteboard, then they can visit macOS system instead of this app.