Dedicated Automagic's data backup, list of flow state

[Desmanto]

Today is a very messed up day. I had a nightmare but somehow solved it since I have backup.

It was Sunday 12:00 noon, my TWRP flow kicks in and I proceed yes to make my weekly backup. Fortunately I have made this backup. After finish, I am thinking to continue taking screenshot for my tasker vs automagic comparison article. I was motivated again, since tasker just change ownership to Joao.

Screenshot haunted by black rectangle
But there is one thing that distract me away previously. Somehow 2 months ago, whenever I take screenshot using Automagic (I make a flow for that), there is a black rectangle covering the Screenshot. It seems to be the statusbar, since it move from above to the bottom, kinda like doing the scanning. As the result, 50% of the screenshot will have this black rectangle covering the screenshot.

Black Rectangle.png (45.97 KiB) Viewed 21552 times

I have to retake the screenshot, sometimes up to 3-4 times to get the perfect screenshot which doesn't have this black rectangle. But I remember I never have that problem before, it just happen within this 2 months. Since I am creating another flow, I simply ignore it.

Today, I decided to solve the problem anyway. I look at my secondary phone and check if it has the same problem. Using action take screenshot, and yes, that black rectangle moving from top to bottom. But it doesn't appear in the screenshot result. Maybe it is because I don't have much flow in this phone, so the process is much faster.

Regress to January
I tried to remember if anything changes and if it has the same configuration with the secondary phone. This black rectangle seems to animate the status bar. Flashing back to January, reminds me about animation. I have to enable Windows animation back so I can open the App HotWord Detection : viewtopic.php?f=4&t=7197 Other users are reporting if the windows animation are off, the app won't open properly.

So I want to enabled it back. But somehow at developer option, those 3 settings are grey-out. I can't change them back. OK, i simply use Automagic Set System setting - global and set all those value back to 1. The animation didn't appear until I restart the phone. Now animation is back to 1x, I can open hotword app now; but still can't change animation speed at developer option. Don't know whats wrong, because I have tried to disable xposed and revert some other setting, still can't change it. I digress, since main purpose was achieved (to open the hotword app properly).

Boldly delete system setting
I leave the animation enabled and for 2 months and I have this black rectangle haunting my screenshots. I am curious again to find the solution so I can change the animation again. I do backup first using Titanium Backup, at Settings and Settings provider (where the setting database is stored). I BOLDLY delete these 3 values using sqlite editor : animation_duration_scale, transition_animation_scale, and window_animation_scale. Since those values correspond to the animation speed in the developer option. I just have TWRP backup an hour ago and do another backup at Titanium Backup, so it supposed to be safe (I can restore if something goes wrong). I think those values will be reset and I can change them again after restart.

I restart the phone, bootstuck! What, how can deleting such values cause bootstuck? (hehehe, of coz, I am playing fire with system setting). Since it is stucked, I copy out the titanium backup from settings provider, extract out the database, using TWRP file manager to copy it back to the original place, replacing the ones that I have deleted. Still stucked Tried to set the permission to 755, still stucked. Ah, I don't want to troubleshoot anymore, just use time machine to get back my phone to 1 hour ago. Restore the whole /data partition using TWRP and my phone boot back normally.

Flow restored, but need to enable back
When I open the automagic, It prompt me something that the flows.xml has been modified, whether to load it up again but disable all flows. Arrgh, I ever face this before during the dev testing. I just realized that I need to stored the flow state of each flows before. Now I have to go thru 100+ flows and enable 40+ of them back. Fortunately I open Automagic everyday, so i still remember most of them. (there should be a way to save the flow state so we can restore the state when restoring flows.xml).

Actually, not long after I made the request about favourite elements : viewtopic.php?f=4&t=7253, I have made another flow to count the statistic from Automagic. So far, I have finish until the Flows, Triggers, Conditions, Actions; and will continue to note, widget, widget's element and action. That's why I know I have 43 flows enabled out of 113 flows. I should have stored the flow state previously to a glovar, so I can loop set flow state back when I restore it. (will add this part later). But I think there should be a function or something to keep the track of all flow state, so We don't have to enable back each flow when we restore or move device.

Nightmare continue...
But wait, nightmare haven't finish. I give up on trying to change the animation speed using developer option. So I simply use set system setting and restart the phone, since I don't change it frequently; why should bother. I tried first at the secondary phone, to check which of these 3 values correspond to the black rectangle. After several test, I know it is the transition_animation_scale (second one). So even if I enable the 1st and 3rd, if I turn off the 2nd option, that black rectangle won't haunting my screenshot anymore. I simply use the set system setting again and set only that value to 0; the other 2 leave it 1x. Since I just restarted, I think it is much faster if I use "soft reboot" option from Xposed Installer. It will only restart the UI.

I tap the "soft reboot", phone turned black and UI reload up, much faster than usual reboot. But somehow automagic not started. So I tap open the Automagic manually to enable the service. Automagic prompt me that the widgets.xml has been modified (similar to flows.xml above) and ask If I want to load it up. Of course I do. But somehow I have some dejavu and bad feeling. I tried some flow to make sure it is working fine. I simply tried Screenshot flow, since I am going to test the black rectangle bar anyway. And horray, the rectangle bar doesn't haunt my screenshot anymore.

Widget action disappeared!!!
But wait a minute, why I can't swipe to remove the widget shown? (which cover the status bar to make it tidy). I swipe it many times and it seems doesn't work (I use swipe left to hide the widget). I had a bad feeling, I quickly browse to the widget and check the clickable action. ALL of clickable actions from all widgets HAVE GONE!!! Arrgh, just now it is flows.xml, now it is widgets.xml. What's wrong with today?

I quickly check the widgets.xml, but I can't see any difference. I remembered I ever experienced the same before. Suddenly my widgets lose all of their actions. But I don't know the cause. This time I know it is the "soft reboot" that kills all of the widget actions, modifying the widgets.xml. Previously I still haven't use widget intensively, so I simply recreate the actions, it is not a big problem. But now, i have so many action mapped to many widgets, It will be so painful to recreate all of them, not to mention If I missed some of the actions. No other way, I must restored that particular widgets.xml from my TWRP backup.

Restoring from TWRP
I am thinking to restore only Automagic's data from Titanium Backup using extract from nandroid. But I will face the same flows.xml problem again. I only need the widgets, so why should I go to the same trouble of enabling all flows again? I googled the way to extract data directly from TWRP backup. (I forgot the method already, since rarely need it). Found out that the backup is just a tar.gz archive, I simply extract out the automagic data folder and take out the widgets.xml. I compared the size, yes it has change. widgets.xml with action still intact is 147 KB, while the one which has lost all of its action is 143 KB. So i just stop automagic service first, copy and replace the widgets.xml and start Automagic again. Same prompt that widgets.xml has been modified outside of automagic, whether to load it up. Yes, absolutely. Immediately check the widget and now all of the actions are back. Ah, what a relief. Test the flow, the widget can be swiped out properly again. Since Automagic reload the widgets.xml fine, it must be the "soft reboot" that kills the actions.

Damn, I should be typing the Automagic vs tasker article today, but I ended documenting this nightmare. (for my own future reference). But at least now I solved the black rectangle box that haunts my screenshot; and also learn something again.

Lesson learnt
What I can learn from my nightmare today :
1. Always backup the flows, widget and glovar. Just backup the whole Automagic main data folder. I seem too relied upon TWRP and Titanium Backup (I like to live dangerously ). I should make a flow that backup it up properly
2. As a side backup, take a note also on which flows are enabled. I am going to create a flow to backup those flows state, since Automagic doesn't have it yet. But won't finish in near future.
3. Never delete the system setting value randomly (unless you have TWRP backup before).
4. Black rectangle box which haunt the screenshot action can be removed by disabling the transition (transition_animation_scale set to 0).
5. Never use "Soft Reboot" (only for root users). It will nuke all your widgets actions. I don't know if it happens only at my phone, but better safe than sorry.

Feature Request
From the nightmare, I would like to request some new features :
1. A dedicated backup feature built-in, not using flow.
So there will be a new option in setting to backup the whole Automagic data, especially : flows, widgets, global variables. A flow to backup the data is not the real solution. Because a flow can be disabled, deleted, replaced, can hit error exception or emergency stop and parallel/stop execution can cause chaos etc. Consider that Automagic need to protect and backup its own data, it supposed to have its own dedicated built-in solution, not leaving it to a flow to backup. I consider Whatsapp chat backup solution is quite neat. So everyday at certain hour (which can be set), Automagic will zip the flows.xml, widgets.xml and variables.bin (glovar) to a separate folder, maybe /sdcard/AMbackup. The zip files will be appended date, so maybe the name will be "/sdcard/AMbackup/AM_backup_20180503.zip". Retain the backup until the last 7 days. The backup is enforced and can't be disabled, except when Automagic service is not running (some users might want to stop Automagic completely for certain period of time). We can set the time to backup and can tap to manual backup. Then the restore function has the choice to restore certain flows, widgets, or certain global variable. Almost like Whatsapp backup/restore method.

2. Option to keep the flow state (the enabled ones) when restoring flows.xml.
Or at least preselect all the enabled flow, prompt us to disable the ones we think might cause problem (maybe automagic crashed by certain flows).

3. Expand the function of getFlowNames() and getFlowGroupNames() so it return a nested Map contains the flow data.
Current getFlowNames() and getFlowGroupNames() only return a list of the flow/group. There is no way to check if the flow is enabled or not. I even made the flow to use the condition "Flow Enabled" to loopcheck all of the flow. But it is too slow, and require a very high AES (as high as twice of the flow number, I have 100+, so at least AES 200+). Later I realize I can use findAll() to the flows.xml and parse the flow property, require only 2 elements. So I extract out the flow property, total there are 7, first one is flow name.
1. name
2. group
3. enabled
4. lastExecutionStartTime
5. lastExecutionEndTime
6. executionPolicy
7. emergencyStopCount

I then create a nested map (map map) from it. First level map key is the whole flow name, second level map key is 6 properties of the flow. The example, one of the flow name is Efaktur, this is the second level map.

Efaktur.png (61.99 KiB) Viewed 21552 times

Two exceptions for group name and emergencyStopCount. When parsing it from the xml, both of this can be not exist there. If group name is null, it is the "ungrouped" one. I replace it using convertNull(). For emergencyStopCount, if it is null, then it is "default" (default is 60 if we don't change it).

Since each flow will have its own complete property, I can then create any new list from this. As long as we have the master data, extracting certain element will be as easy as looping. Example, I need to keep the flow name which state is enabled. I simply use this script.

Code: Select all

flowenabled = newList();
for(i in getMapKeys(flowmap))
{
  if(i["enabled"] == true)
    addElement(flowenabled, i);
}
global_flowenabled = flowenabled;

flowenabled will contains all enabled flows, basically capture all enabled flow state. When restoring flows.xml, Automagic disabled all the flow. I can then use this global_flowenabled in comma limited format in the Set Flow State to enabled back all the captured enabled flow state, no need to enable it back one by one. Of course the list should be checked first against available flows, since maybe the new flows.xml restored don't contain the enabled flows previously.

So getFlowNames() can be expanded to map map object, where the first level key is the flow name, second level is 6 flow's properties. Optionally, getFlowGroupNames() can be expanded to be map map map object (3 level), 1st level group name, 2nd level flow name, 3rd level 5 flow's propreties. But getFlowGroupNames() is redundant, since we can create it from the getFlowNames() if it a map map object already.

4. Add a help text in the take screenshot and screen recording action
that transition_animation_scale can cause a rectangle black box being captured. (I don't know if it happens in all devices, need to test it).

I know this will take a long time to implement, so in mean time I just use my own workaround : create the backup flow, save the flow state and parse my owns flow map.

Thank you for reading so far. Just another day of my long troubleshooting.

Regards,
Desmanto

[yxd0018]

will copying the whole data folder be considered full backup?
Wonder if there is a way to automate TWRP backup.

[Desmanto]

Yes, the whole Automagic folder. Better zip it also, to make it more compact and single file only. You can use my Automagic Backup flow for the flow and some supporting files : viewtopic.php?f=3&t=7858

I also have semi auto TWRP backup on sunday, including to keep only the last 2 backups. It give notification on Sunday 12:00, and there is a way to postpone it if I am still using the phone. Can't make it full automatic, as It might disturb my activities. Can't make it at night/unattended backup too, since there is a risk that the backup failed and stuck at TWRP. I thought I am the only one thinking to use this. I've never shared the TWRP flow yet.

[yxd0018]

Thanks. I can integrate your backup flow into my google drive backup flow.
Can you share your TWRP backup flow? As the TWRP backup is pretty big, where do you store it?

[yxd0018]

Separate question, when you backup with TWRP, is the single system backup file takes 6GB? How do you extract img file or copy back to phone from pc? I kinda stuck here.

[Desmanto]

Planned to do so. But I will be on vacation tomorrow until next week. As you can see from most of my flow sharing, I usually prepared it perfectly, including how to use it. It takes some time to prettify the flow and type the post.

I still store the TWRP backup at the internal storage. Once in a while, sometimes weeks/months, I copy it to PC as backup. I only keep last 2 backups, each is around 4,5 GB, compressed (uncompressed can takes up to 9 GB). I only backup /data, not /system anymore. As I use custom ROM and I can easily flash the ROM if needed.

I don't extract the img file anymore, I simply copy it manually using MTP or wifi SMB share. I rarely need to restore it, only on a very special occassion. Usually it is just weekly backup. If I am not wrong, You can extract the backup using peazip or 7zip and its kind. I forgot already, I will try it again later when typing the post.

[yxd0018]

I backed up every single partition with twrp. When I try to restore, it always failed in system image. I find out I don't even need system image nevertheless backing it up.

[Desmanto]

Just backup "system", not "system image". But I never backup system anymore, since it is as easy as flashing the ROM + gapps, rather than backing it up every week. I don't need to restore system, so it is more efficient not to backup it up. Only if I use phone which is very difficult to find stock ROM, then I will backup the system too.

I still need to prettify my TWRP flow first and test run it next week.

[yxd0018]

Yeah, I found out that as well. I put the initial backup of all partitions except cache, system, system image to the cloud. Then probably only backup data later on. It's quite a learning experience for me. Thanks for the helping.