W1CDN/aprs_tool
0
0
Fork 0
Code Issues 45 Pull Requests Projects Releases Activity

Sort, filter, and limit results at /packets #23

Merged
W1CDN merged 14 commits from sort-filter into main 2023-05-14 10:18:10 -05:00
Conversation 13 Commits 14 Files Changed 6 +129 -83
W1CDN commented 2023-04-16 18:51:41 -05:00
Owner
Copy Link

Closes #15, #3.

Closes #15, #3.
W1CDN added 2 commits 2023-04-16 18:51:41 -05:00
W1CDN commented 2023-04-16 19:26:32 -05:00
Author
Owner
Copy Link

I wonder if using aprs_api/packets/<callsign-ssid> works better on the back end than aprs_api/packets?from=<callsign-ssid>.

Also, this needs to happen on the query side, so we don't pull all rows and then cut them down every single time:

# Limit to number of records requested
        data = select_all_frames(conn)[-n:]

Should probably run everything through the query anyway.

I wonder if using `aprs_api/packets/<callsign-ssid>` works better on the back end than `aprs_api/packets?from=<callsign-ssid>`. Also, this needs to happen on the query side, so we don't pull all rows and then cut them down every single time: ``` # Limit to number of records requested data = select_all_frames(conn)[-n:] ``` Should probably run everything through the query anyway.
mattbk added 1 commit 2023-04-16 19:29:15 -05:00
mattbk added 1 commit 2023-04-16 21:04:34 -05:00
W1CDN commented 2023-04-16 21:09:04 -05:00
Author
Owner
Copy Link

It might be time to use sqlalchemy or some other way to abstract the database calls. My first pass is pretty ugly.

For example, I should be able to arbitrarily WHERE on any field.

It might be time to use sqlalchemy or some other way to abstract the database calls. My first pass is pretty ugly. For example, I should be able to arbitrarily WHERE on *any* field.
mattbk added 2 commits 2023-04-23 21:13:39 -05:00
W1CDN commented 2023-04-23 21:25:18 -05:00
Author
Owner
Copy Link

So weird. I uploaded the files to the digi machine and now it shows packets from 2023-04-16.

So weird. I uploaded the files to the digi machine and now it shows packets from 2023-04-16.
W1CDN commented 2023-04-25 12:27:25 -05:00
Author
Owner
Copy Link

The packets are still getting into the db, it's the query that is broken. Will have to rethink this when I have a moment.

The packets are still getting into the db, it's the query that is broken. Will have to rethink this when I have a moment.
mattbk added 1 commit 2023-04-25 14:19:38 -05:00
W1CDN commented 2023-04-25 14:34:07 -05:00
Author
Owner
Copy Link

The query was ordering by row ID by default, I added ORDER BY created DESC.

Now messages don't work (unrelated to recent change):

{'raw': 'W1CDN-2>APQTH1,K0UND-2,WIDE2-2::W1CDN-2  :Test msg{01}', 'from': 'W1CDN-2', 'to': 'APQTH1', 'path': ['K0UND-2', 'WIDE2-2'], 'via': '', 'addresse': 'W1CDN-2', 'format': 'message', 'message_text': 'Test msg', 'msgNo': '01', 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}
Error with SQLite!

Missing msgNo field.

The query was ordering by row ID by default, I added ORDER BY created DESC. Now messages don't work (unrelated to recent change): ``` {'raw': 'W1CDN-2>APQTH1,K0UND-2,WIDE2-2::W1CDN-2 :Test msg{01}', 'from': 'W1CDN-2', 'to': 'APQTH1', 'path': ['K0UND-2', 'WIDE2-2'], 'via': '', 'addresse': 'W1CDN-2', 'format': 'message', 'message_text': 'Test msg', 'msgNo': '01', 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} Error with SQLite! ``` Missing `msgNo` field.
W1CDN commented 2023-04-27 09:30:57 -05:00
Author
Owner
Copy Link
{'raw': 'K0UND-2>APDW16,WIDE2-1:!b69Ld5xIC#  #UND SARA; undsara.org; 146.865 (-) 123.0', 'from': 'K0UND-2', 'to': 'APDW16', 'path': ['WIDE2-1'], 'via': '', 'messagecapable': False, 'format': 'compressed', 'gpsfixstatus': 0, 'symbol': '#', 'symbol_table': 'b', 'latitude': 47.92433438515617, 'longitude': -97.067666685918, 'comment': 'UND SARA; undsara.org; 146.865 (-) 123.0', 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}
{'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}
Traceback (most recent call last):
  File "kiss_and_db.py", line 128, in <module>
    main()
  File "kiss_and_db.py", line 100, in main
    a['path'] = str(a['path'])
KeyError: 'path'

Why does this look like two dicts? Oh, the first part may have gone in fine, and this one is the problem: {'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} Was there an empty/unparsed frame that happened?

``` {'raw': 'K0UND-2>APDW16,WIDE2-1:!b69Ld5xIC# #UND SARA; undsara.org; 146.865 (-) 123.0', 'from': 'K0UND-2', 'to': 'APDW16', 'path': ['WIDE2-1'], 'via': '', 'messagecapable': False, 'format': 'compressed', 'gpsfixstatus': 0, 'symbol': '#', 'symbol_table': 'b', 'latitude': 47.92433438515617, 'longitude': -97.067666685918, 'comment': 'UND SARA; undsara.org; 146.865 (-) 123.0', 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} {'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} Traceback (most recent call last): File "kiss_and_db.py", line 128, in <module> main() File "kiss_and_db.py", line 100, in main a['path'] = str(a['path']) KeyError: 'path' ``` Why does this look like two dicts? Oh, the first part may have gone in fine, and this one is the problem: `{'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}` Was there an empty/unparsed frame that happened?
W1CDN commented 2023-04-27 09:45:33 -05:00
Author
Owner
Copy Link

On live code, moved the rest of the processing inside the try()...else() at

kiss_and_db.py Line 91 in f396fe87af
try:
.

On live code, moved the rest of the processing inside the try()...else() at https://amiok.net/gitea/W1CDN/aprs_tool/src/commit/f396fe87af0288e6e57810065945dfb6cec7e45c/kiss_and_db.py#L91.
mattbk added 1 commit 2023-04-27 19:19:25 -05:00
mattbk added 1 commit 2023-04-27 19:29:29 -05:00
W1CDN commented 2023-04-28 10:12:38 -05:00
Author
Owner
Copy Link
{'raw': 'K0UND-2>APDW16,WIDE1-1,WIDE2-1:}WLNK-1>APWLK,TCPIP,K0UND-2*::W1CDN-13 :ack1', 'from': 'K0UND-2', 'to': 'APDW16', 'path': ['WIDE1-1', 'WIDE2-1'], 'via': '', 'format': 'thirdparty', 'subpacket': {'raw': 'WLNK-1>APWLK,TCPIP,K0UND-2*::W1CDN-13 :ack1', 'from': 'WLNK-1', 'to': 'APWLK', 'path': ['TCPIP', 'K0UND-2*'], 'via': '', 'addresse': 'W1CDN-13', 'format': 'message', 'response': 'ack', 'msgNo': '1'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}
Error with SQLite!

I wonder if the curly braces in the frame are causing problems.

``` {'raw': 'K0UND-2>APDW16,WIDE1-1,WIDE2-1:}WLNK-1>APWLK,TCPIP,K0UND-2*::W1CDN-13 :ack1', 'from': 'K0UND-2', 'to': 'APDW16', 'path': ['WIDE1-1', 'WIDE2-1'], 'via': '', 'format': 'thirdparty', 'subpacket': {'raw': 'WLNK-1>APWLK,TCPIP,K0UND-2*::W1CDN-13 :ack1', 'from': 'WLNK-1', 'to': 'APWLK', 'path': ['TCPIP', 'K0UND-2*'], 'via': '', 'addresse': 'W1CDN-13', 'format': 'message', 'response': 'ack', 'msgNo': '1'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} Error with SQLite! ``` I wonder if the curly braces in the frame are causing problems.
mattbk added 1 commit 2023-05-13 11:10:25 -05:00
W1CDN commented 2023-05-13 11:11:15 -05:00
Author
Owner
Copy Link

Excited about the latest commit, which allows you to query on any field in the frames table.

I'm not sure if you can do wildcards or not.

Excited about the latest commit, which allows you to query on any field in the frames table. I'm not sure if you can do wildcards or not.
mattbk added 1 commit 2023-05-13 11:14:25 -05:00
W1CDN commented 2023-05-13 11:16:35 -05:00
Author
Owner
Copy Link

I added LIKE instead of =, which will let you do wildcards with % as in:

  • http://127.0.0.1:5001/packets?created=2023-04% or
  • http://127.0.0.1:5001/packets?from=%UND%.

https://www.sqlitetutorial.net/sqlite-where/

I added `LIKE` instead of `=`, which will let you do wildcards with `%` as in: - `http://127.0.0.1:5001/packets?created=2023-04%` or - `http://127.0.0.1:5001/packets?from=%UND%`. https://www.sqlitetutorial.net/sqlite-where/
mattbk added 1 commit 2023-05-13 17:26:44 -05:00
W1CDN commented 2023-05-13 17:41:00 -05:00
Author
Owner
Copy Link

To close #27, is it easier to query on timestamp or by the text version of created?

timestamp actually doesn't seem to make sense. It converts about two days off. Is it what I think it is?

Since created is stored as text, options are

  • save a second field as Unix seconds, which could be queried with comparators (greater than, less than, equal to)
    • requires user to calculate Unix seconds to use in query
  • allow query of string datetime with comparators
    • requires special treatment of created
    • requires parsing both db created and query string into Unix (maybe with DATETIME in the db query itself?)

What's the use case for this, anyway? I think returning Unix time like https://aprs.fi/page/api and letting the downstream application worry about comparators is fine.

You can also already pull limited time blocks using wildcards on created: http://127.0.0.1:5001/packets?created=2023-04% returns packets in April 2023.

In any case, the next thing should be

  • add field for created_unix field in db frames table

Ugh, because created is automatic in the row creation, a direct conversion would mean two more db calls (one to read created, one to write to second column). Probably better to calculate created_unix when writing the row originally.

To close #27, is it easier to query on `timestamp` or by the text version of `created`? `timestamp` actually doesn't seem to make sense. It converts about two days off. Is it what I think it is? Since `created` is stored as text, options are - save a second field as Unix seconds, which could be queried with comparators (greater than, less than, equal to) - requires user to calculate Unix seconds to use in query - allow query of string datetime with comparators - requires special treatment of `created` - requires parsing both db `created` and query string into Unix (maybe with `DATETIME` in the db query itself?) What's the use case for this, anyway? I think returning Unix time like https://aprs.fi/page/api and letting the downstream application worry about comparators is fine. You can also already pull limited time blocks using wildcards on `created`: `http://127.0.0.1:5001/packets?created=2023-04%` returns packets in April 2023. In any case, the next thing should be - [x] add field for `created_unix` field in db frames table Ugh, because `created` is automatic in the row creation, a direct conversion would mean two more db calls (one to read `created`, one to write to second column). Probably better to calculate `created_unix` when writing the row originally.
mattbk added 2 commits 2023-05-13 20:59:42 -05:00
W1CDN commented 2023-05-13 21:07:45 -05:00
Author
Owner
Copy Link

New field created_unix works. As a bonus, you can group vaguely by timestamp using a wildcard. For example, http://127.0.0.1:5001/packets?created_unix=1684029% captures values between 1684029000 and 1684029999 (between Sat May 13 2023 20:50:00 and Sat May 13 2023 21:06:39 GMT-0500).

(The same wildcard-tail search will return different results once Unix time gains another order of magnitude, but that's not until 2286.)

New field `created_unix` works. As a bonus, you can group vaguely by timestamp using a wildcard. For example, `http://127.0.0.1:5001/packets?created_unix=1684029%` captures values between 1684029000 and 1684029999 (between Sat May 13 2023 20:50:00 and Sat May 13 2023 21:06:39 GMT-0500). (The same wildcard-tail search will return different results once Unix time gains another order of magnitude, but that's not until 2286.)
W1CDN commented 2023-05-13 21:13:19 -05:00
Author
Owner
Copy Link

A lot of this work can be repurposed for #28, but I'll open a new PR for that.

Letting this branch run on the live system for a bit to see if anything breaks, then will close.

A lot of this work can be repurposed for #28, but I'll open a new PR for that. Letting this branch run on the live system for a bit to see if anything breaks, then will close.
W1CDN merged commit 4203af5b81 into main 2023-05-14 10:18:10 -05:00
W1CDN deleted branch sort-filter 2023-05-14 10:18:20 -05:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: W1CDN/aprs_tool#23
No description provided.
Delete Branch "sort-filter"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?