Manipulate UI

[Bluscre]

I would like to have a way to not only control but to modify/manipulate the graphical user interface of an app. On windows that isn't so easy because most apps don't rely on common frameworks like .NET
But on android it *shouldn't* be too hard AFAIK. IDK what much of permissions would be needed ( I would be okay with root+xposed cause i have them on all my devices anyway )

What i was thinking of:

• body = getCurrentBody() // Should return the main background of the current view so you can manipulate its color for example.
• element = getElement/s(str Text)
• element = getElement/sByID(str ID)
• button = newButton(str Text, str FontColor, str BGColor, str BGImage) [?]
• newButtonFunc(button, execFlow, "button pressed") [?]
• label = newLabel(str Text, str FontColor)
• removeElement(element)
• replaceElement(element, replacement)
• addElement(parent, newElement)
• setElementBG(body, "#000000")
• resizeElement(element, height, width)
• moveElement(element, x, y)

[Desmanto]

Seems like you are talking about Action Control UI. AFAIK, Control UI only allow action so far accessibility is involved. Google won't allow any other app to manipulate ohter app's UI directly from the core element, it will be tapjacking issue (I remember there are still several exploits available). Accessiblity only can change as much as changeable field, such as input box, checkbox, radio button etc.

To change from the core UI element, require root and xposed. It require to create a compatible xposed module, which take a lot of time. I would prefer this time invested to other more generic feature which can be use by most user. There are a lot of requests awaiting.

body >> don't know if possible. But if only to retrieve the color, there are function getPixelColor() and several others (search "color" at function)
element by Str text and ID >> can be done via Control UI already. But the element must be exposed to Accessibility
button, newButtonFunc, label >> can't add new button/label within app. Use show custom widget instead
remove, replace, add, set, resize, move element >> can't be done as well. But you can use widget, change the view similiar to the main app dan place it in front of them to virtually make them appear to be changed.

Maybe Martin can explain more which can be done or not. Or alternative on Xposed module.

[Bluscre]

(I remember there are still several exploits available).

Can't standalone AM offer to use these?

To change from the core UI element, require root and xposed.

( I would be okay with root+xposed cause i have them on all my devices anyway )

body >> don't know if possible. But if only to retrieve the color, there are function getPixelColor() and several others (search "color" at function)

I don't want to get the body background color, i want to change it. Some apps don't give the user a option to select a dark theme (like youtube for example) so i want to override it's background color with AM.

element by Str text and ID >> can be done via Control UI already. But the element must be exposed to Accessibility

I want to store the element itself in a variable, manipulating the element from there would give so much possibilites (add new events for buttons, labels, etc (onClick, onLongClick, onHover))

button, newButtonFunc, label >> can't add new button/label within app. Use show custom widget instead
remove, replace, add, set, resize, move element >> can't be done as well. But you can use widget, change the view similiar to the main app dan place it in front of them to virtually make them appear to be changed.

Custom widgets is exactly what i don't want to use, they look ugly are overcomplicated and bloat a flow completely

[Desmanto]

(I remember there are still several exploits available).

Can't standalone AM offer to use these?

Sorry, I am not clear with the exploit. I mean the tapjacking exploit, which use similiar concept to custom widget, by putting the widget on top of the element but still passing the tap to the element on the bottom. It uses accessibility feature. So not the modifying element that you want.

To change from the core UI element, require root and xposed.

( I would be okay with root+xposed cause i have them on all my devices anyway )

Yeah, it is about the time and priority. Creating xposed module require a lot of work. As you know that xposed development also quite slow, rovo89 also takes a lot of time and today we still don't have official xposed framework (there are a lot of unofficial unstable version) for Nougat yet (Oreo is out already).

I don't want to get the body background color, i want to change it. Some apps don't give the user a option to select a dark theme (like youtube for example) so i want to override it's background color with AM.

It is probably easier to find a modified apk with dark theme, than waiting for this feature to be implemented. Besides Xposed, modifying this background or any element directly need direct access to the apk, which is similiar to patching or modding. I don't think Automagic want to do something like that and get banned by Playstore. Everyone lose then.

I want to store the element itself in a variable, manipulating the element from there would give so much possibilites (add new events for buttons, labels, etc (onClick, onLongClick, onHover))

For getting the whole element ID, I agree. There is no function to retrieve all available element ID in current windows, only can get the text. But after you get them all, you still can't do anything to it without xposed or direct apk mod. It only helps in determining with element need to be clicked or set the text, etc.

Custom widgets is exactly what i don't want to use, they look ugly are overcomplicated and bloat a flow completely

Unfortunately, this is the only possible solution provided by accessibility.