getText method in Control UI sometimes returns empty string

[EghamQin]

Hi Martin,

I use getText extensively to get the text of screen elements.

When I repeatedly run a flow containing getText from Control UI, I find the getText method occasionally returns empty string or null, even thought the screen elements do contain texts, which can be detected by Overlay Control.

I wonder if it is a bug or inappropriate usage of the method.

For more details, the follows is the part of Control UI action that contains getText method that I use.

Control UI:
details_Y = fromAndToAddress_Y + details_Diff_Y;
detailsText = getText (details_X, details_Y);

agentName_Y = fromAndToAddress_Y + agentName_Diff_Y;
agentNameText = getText (agentName_X, agentName_Y);

postedJobNumber_Y = fromAndToAddress_Y + postedJobNumber_Diff_Y;
postedJobNumberText = getText ( postedJobNumber_X, postedJobNumber_Y);


In the above example, agentNameText = getText (agentName_X, agentName_Y) and postedJobNumberText = getText ( postedJobNumber_X, postedJobNumber_Y) occasionally return empty string or null.

Wish you would shed some lights on the obscure problem.

Many thanks,
Qin

[Martin]

Hi,

This should not happen when the elements containing the text are steadily showing on screen and no other 'window' like a popup or a menu are displaying on screen. Maybe the script in action Control UI is executed too soon. Did you try to add a short sleep(500) or similar before the script is executed?

Under what situation does this problem occur? Can you describe the steps to reproduce this problem in more detail or send me a simple flow to reproduce this problem?

Regards,
Martin

[EghamQin]

Hi Martin,

Thank you for your reply.

I have tried to add sleep(500) just before the getText method. But it does not solve the problem. The getText still occasionally returns empty string.

At the moment, I am not able to deterministically reproduce the bug. It happens fairly occasionally (around 1 in 10 times). I have attached the flow to this post, if you would like to do a code review.

Please notice that this flow cannot be run on its own as it requires variables from other flows. Besides, this flow is designed only for a target app, whose owner does not allow me to disclose the app.

Kind regards,
Qin

[Martin]

Hi,

Maybe the accessibility feature does not work absolutely reliable on the device or there's some side effect of the other app that leads to the failure (animation not finished, a slow network connection, a popup menu of some sort that blocks access to the element etc.).
You could try to workaround the problem by calling getText a second time when the function returned null the first time, maybe add a little sleep in between. Something like this:

Code: Select all

text = getText(...);
if (text==null)
{
  sleep(500);
  text = getText(...);
} 

Regards,
Martin

[EghamQin]

Hi Martin,

Thanks very much for your help.

I have implemented the work-around that you suggested. Unfortunately it seems not solving the problem.

I have been thinking about the possible reasons that you suggested in your post. The details are as follows:

1. animation not finished: the target app does not have any animation. The screen is steady and not moving. So this seems not the problem.
2. a slow network connection: this is likely to happen. But I don't understand how a slow network connection will result in getText not being able to working properly. Would you please explain a little bit more about it?
3. a popup menu of some sort that blocks access to the element etc.: this is very unlikely to happen to the target app, especially at relatively high frequency.

Would appreciate if you could kindly explain about the slow network connection factor. If it is the main reason then it means I have to upgrade the network connection, which is quite a work and expense to me. But I am happy to do it. I just want to understand it more before I take action.

Many thanks,
Qin

[Martin]

Hi Qin

The slow network connection was just a wild guess. Some apps load information from a remote server to show elements like buttons and texts etc. The text of the element might be missing for a second or two when the text is loaded from a remote server and the connection is slow. Maybe this is not the case with the app you use.

Could you please turn on Debug Log and Log to SD card in the preferences of Automagic and send the log to me when this problem occurred the next time?

Could you please also add some log lines to the script to verify that the determined coordinates are correct. Something like this should work:
log("agentName_X: {agentName_X}, agentName_Y: {agentName_Y}");
log("postedJobNumber_X: {postedJobNumber_X}, postedJobNumber_Y: {postedJobNumber_Y}");

Regards,
Martin