Control UI count

[slideman.v]

Hello.
How to count how many times element

Code: Select all

existsElementById etc...

exists???
Second. Any simple way to index elements and select second from the bottom or 6 from the top etc....?
Thx for your experience.

[Desmanto]

I wanna know how to do so too. So far, there is no way to count the element, neither the way to select element no 6 from a list which has the same elementId. I don't know if this is the limitaion of accessibility or simply limitation from Automagic's Control UI.

The workaround, which doesn't work at all time : Use getTextInActiveWindow(), split it using \n (newline). Then search using the text got from getTextById() of the first element ID. That will be the first index of the same element. Then get the 6th element of it, get the text and do whatever needed, example click(pattern) using the text from 6th element.

[Martin]

Hi,

There's currently no way to count the elements or interact with the nth element on screen. You could try to scan the screen
I could provide a function to count the elements with a certain ID but I don't yet understand how you would use it. Could you describe a scenario where you would want to use this feature?

Regards,
Martin

[Desmanto]

Hi Martin,

The use case is mostly with dynamic content, static element with possible dynamic content nearby and multiple command in the same element Id.

Dynamic Content
The apps are something like chatting app, webmobile app, file manager and the kind. In chatting app, such as whatsapp, if I want to retrieve the last message, i probably use

Code: Select all

text = getTextById("com.whatsapp:id/message_text");

However, because all chat use the same element Id, this will retrieve only the first id/message_text from the visible window above. So if we can count all elements with the same Id, we can use index to access the last element. The same goes for other element Id, such as name_in_group_tv which store the the name/number, pushname_in_group_tv which store the name (if the number haven't been save to contact).

One of the user in other forum ask about the possibility to log/act on the chat for certain contact. Using notification on statusbar, it only log for the incoming chat. But when we use control UI to reply to it, we are inside the chat window. Notification won't appear anymore, so the only way is using Control UI. And if we reply to the message, our sent message also not appear in notification. That's why we need to have a way to count all element with the same Id, and then retrieve any message in the correct order. So we can reconstruct the whole chat windows in the log.

I remember someone wants to retrieve the last message from google assistant too, but can't do it because of the same element Id too

Other method that I have been thinking is for remote command and possibly anti theft. We can trigger the first command from notication and open up the chat window. Show overlay to cover the activity and send another keyword as the set of command. Basically it is SSH over Whatsapp. It is only possible if we can get the last message. As if we sent too many message, whatsapp will group it become something like "5 new messages" in notification, making it impossible to get the message itself.

Possible dynamic content
For element with possible dynamic content, I have some flow in the past, to do renaming/numbering file in EFE. The command buttons has 5 command with the same Id, so can't use clickById(), I should use click("Refresh") instead. However, there is a possibility, one of the file/folder can have the same name "Refresh", and the script will click this instead, causing logic flaw, but no error. Later because of the slower response using the UI, and also I don't use EFE anymore; I just find another workaround to auto numbering the file (but haven't created it yet, only concept)

Multiple command in same element Id
But the command in the same element Id still can cause problem in other places. My phone has built-in firewall, called network control. There is a bug in the setting which supposed to set all new apps to blocked, but it doesn't work all the time. So i still have to go to the network control setting and check if it doesn't blocked yet, I should block it manually. I can use getTextInWindow(), coupled with ScrollForwardByid() to scroll find the app name. All apps are located in the same element Id, com.android.networkcontrol:id/app_name. If it is the app name I need to click, I can simply use click(appname), bypassing the clickById() limitation.

Screenshot_2018-02-05-09-32-09_Half_Clean.png (94.62 KiB) Viewed 22482 times

But the problem is it is the radio button at the data and wifi those need to be unchecked. Both these have the same element Id throughout the whole app list. And worse is they don't have label that can be clicked.
Data = com.android.networkcontrol:id/img_btn_right_3g
wifi = com.android.network:id/img_btn_left_wifi
UncheckById won't work properly, as it will only unchecked the first element Id from the top. uncheck(x,y) also won't work, as I can't determine the correct x,y. uncheck(pattern) don't appear at all, as they don't have label. If I can have element Id count, I can list all element with the same Id, upon the app name, data and wifi. So example, I will have 8 elements of app_name, will be populated as
app_name[0] = Application Inspector
app_name[1] = Automagic Premium
.....

The data and wifi don't have name, but no problem, I just need to know they are img_btn_right_3g[0] until img_btn_right_3g[7] and img_btn_left_wifi[0] until img_btn_left_wifi[7]. I can then loop check if the app_name == "Browser", then i is the index i need to uncheck. Use it to uncheck element Id at i.

There are actually other apps which show same element Id in some of the command/menu. Automagic itself also has some same Element Id with different command. I usually just use click(pattern) to overcome the problem. But for other app which doesn't always has label in the button, we can't use click(pattern). Don't know if it is possible to implement, that's why I just deal with it.

Regards,
Desmanto

[slideman.v]

Hi,

There's currently no way to count the elements or interact with the nth element on screen. You could try to scan the screen

i have a solution but looking for something simple. Yeeaaa i use graphic method with compare elements.

[alan23]

Show overlay to cover the activity and send another keyword as the set of command

[Martin]

@Desmanto: Thanks for the great explanation! I'll add it to the todo-list and will think about some ways to improve the situation. I'll wait a few weeks with implementing features in the accessibility area until I get a response from Google if the accessibility features have to be removed.

Regards,
Martin

[Desmanto]

Hi Martin,

I am back to this topic. I am typing the Control UI guide, so I give AutoInput another chance to prove itself. But as expected, It failed to impress me again. After using Control UI for 9 months, Autoinput is too awkward to use. Selecting each element require us to go back thru Automagic for each action. Can't imagine how long I should spend to create a functional flow. But there is one thing It can do (that Automagic can't yet). It is about this thread topic too, the UI element list.

Input Dialog with duplicate label
I create a Input dialog multiple choice, have 5 elements. There are two elements have the same label : "hello". Then I use Autoinput to select the second hello (4th element). After selecting and go back to Automagic, at the Select Element Field, Autoinput give me 4 choices. (before I choose Click event)

1. Can select List.png (135.37 KiB) Viewed 22017 times

Element Id, Element Text and Element Center Point (x,y) already available in Automagic. But that Element number 4 in list 'android:id/select_dialog_listview' is the one doesn't exist in Automagic. Using Autoinput I can select the second 'hello' from the list, by simply selecting Element number 4 from the select_dialog_listview.

I tried to look more in the configuration.

2. List Type Id 4.png (243.51 KiB) Viewed 22017 times

Autoinput has Field Type : Text, Id, Focus, List, and Point. Text, Id, Point already available in Automagic. Focus available in some element and can be changed manually after pasting the script. But we don't have Field Type List (UI element list) in Automagic. It is because Autoinput has it, that's why it can pick Field Id 4, which is to select 4th element from the listview. The right side is the result after I show up available variables from Autoinput.

Automagic Control UI
I tried the same thing in Automagic, this is the result.

3. Automagic can't Select List element Id.png (205.04 KiB) Viewed 22017 times

From middle one, if I tap on the 4th element (second 'hello'), then it will pop up choices as shown on the left; where we have click(pattern), click(x,y) and clickById(Id). But we don't have clickListId(listviewId, index) >>> I made up the name for example only. Using check() or click() works the same for this Input dialog.

For the listview, I can see a bigger rectangle covering the whole list. If I tap on the narrow spaces at the top or bottom, it will pop up choices as shown on the right. There I can find clickById("android:id/select_dialog_listview"). But if I use this, nothing happen. I assume we can use clickById("android:id/text4"), but that doesn't work. AFAIK, Id text1 is provided by the app UI element Id, not from Automagic. (well, in this case, the Input Dialog is from Automagic itself).

Automagic Limitation
Because of this limitation, I can't select the second "hello" (4th element). Anytime I have duplicate label, I can't select or interact with the second one. If I use click("hello"), it will only select the first hello it encounter from top right of the screen. If I use clickById("android:id/text1"), it also only select the 1st element (even if it is not "hello"). If I use click(540, 1109), I doomed myself to a logic flaw. Because if by any chances the 1st to 3rd element now have longer text and covering more spaces, the click(x,y) won't work anymore. Worse is it can click the wrong element, causing havoc. Click(x,y) also doesn't work anymore if I change my phone UI dpi, causing the element size to contract or expand.

Solution
So the only solution if we can have clickListid(listviewId, index). In this case

Code: Select all

clickListId("android:id/select_dialog_listview", 3); //choose index 3 (4th element) from list Id "select_dialog_listview"

Of course, this doesn't limited to click only, it also expand to the other function such as longclick, check, getText and etc.

Hope this is clear enough to implement it when you have the time to look at it.

Thank & Regards,
Desmanto