W1CDN/aprs_tool
0
0
Fork 0
Code Issues 46 Pull Requests Projects Releases Activity

Add background KISS connection, log frames to database #20

Merged
W1CDN merged 10 commits from add-kiss into main 2023-04-16 16:45:54 -05:00
Conversation 15 Commits 10 Files Changed 9
+348 -5
W1CDN commented 2023-04-13 17:19:35 -05:00
Owner
Copy Link
Copy Source

Closes #5.

Closes #5.
W1CDN added 1 commit 2023-04-13 17:19:36 -05:00
Stub out a KISS connection. 5e86b40f38 
It seems to simultaneously run the API and keep the KISS connection open 
in the background.
W1CDN commented 2023-04-13 20:30:03 -05:00
Author
Owner
Copy Link
Copy Source

https://stackoverflow.com/a/7224186

https://stackoverflow.com/a/7224186
W1CDN commented 2023-04-13 20:46:10 -05:00
Author
Owner
Copy Link
Copy Source

I think I should be using https://github.com/python-aprs/aprs3.

For versions of the KISS transports which do NOT automatically encode/decode APRS data, see kiss3.

I think I should be using https://github.com/python-aprs/aprs3. > For versions of the KISS transports which do NOT automatically encode/decode APRS data, see kiss3.
mattbk added 2 commits 2023-04-13 21:11:08 -05:00
Snapshot. b4bc632ded
Switch to aprs3 package. 4f83f76bf5
W1CDN commented 2023-04-13 21:12:20 -05:00
Author
Owner
Copy Link
Copy Source

Just barely working now (packet into database), but I don't understand the structure of the APRSFrame object, so it's going in as a string right now.

> repl(frame)
APRSFrame(destination=Address(callsign=b'APZ001', ssid=0, digi=False), source=Address(callsign=b'W1CDN', ssid=0, digi=False), path=[Address(callsign=b'K0UND', ssid=2, digi=True)], control=Control(v=b'\x03'), pid=b'\xf0', info=StatusReport(raw=b'>Hello World!', data_type=<DataType.STATUS: b'>'>, data=b'Hello World!', comment=b'', status=b'Hello World!', _timestamp=None))

Switching to aprslib, which parses more easily?
https://aprs-python.readthedocs.io/en/stable/examples.html

Just barely working now (packet into database), but I don't understand the structure of the APRSFrame object, so it's going in as a string right now. ``` > repl(frame) APRSFrame(destination=Address(callsign=b'APZ001', ssid=0, digi=False), source=Address(callsign=b'W1CDN', ssid=0, digi=False), path=[Address(callsign=b'K0UND', ssid=2, digi=True)], control=Control(v=b'\x03'), pid=b'\xf0', info=StatusReport(raw=b'>Hello World!', data_type=<DataType.STATUS: b'>'>, data=b'Hello World!', comment=b'', status=b'Hello World!', _timestamp=None)) ``` Switching to aprslib, which parses more easily? https://aprs-python.readthedocs.io/en/stable/examples.html
W1CDN commented 2023-04-13 21:37:08 -05:00
Author
Owner
Copy Link
Copy Source

Next up,

  • move code from tcp_kiss_send_recv.py to kiss_and_db.py.
  • test on SBC to see if there is a ton of load from the subprocess

Now need to know all the possible fields that might come up, so they can be added to the db (or maybe write a function to add new fields on the fly?).

Are fields listed in the spec? I think so, see here: https://aprs-python.readthedocs.io/en/stable/parse_formats.html

image

**Next up**, - [x] move code from `tcp_kiss_send_recv.py` to `kiss_and_db.py`. - [x] test on SBC to see if there is a ton of load from the subprocess Now need to know all the possible fields that might come up, so they can be added to the db (or maybe write a function to add new fields on the fly?). Are fields listed in the spec? I think so, see here: https://aprs-python.readthedocs.io/en/stable/parse_formats.html ![image](/attachments/68d13e62-7b38-4e45-801e-8d83b4077673)
image.png
281 KiB
mattbk added 1 commit 2023-04-13 21:38:55 -05:00
Barely parse a frame. b06b87fe4e
W1CDN commented 2023-04-14 12:24:00 -05:00
Author
Owner
Copy Link
Copy Source

Subprocess was running all night and doesn't seem to have broken anything.

Subprocess was running all night and doesn't seem to have broken anything.
W1CDN commented 2023-04-14 19:55:01 -05:00
Author
Owner
Copy Link
Copy Source

Added a more complete list of fields that may come out of aprslib.parse() to the database schema.

How to insert the results of that function into the database without having all the columns represented each time?

Seems like we need to keep around a list of columns, drop it into the statement, but also add null dict items to the dict that comes out of aprslib.parse().

https://stackoverflow.com/a/16698310

values = {
    'title':'jack', 'type':None, 'genre':'Action', 
    'onchapter':None,'chapters':6,'status':'Ongoing'
}
cur.execute(
    'INSERT INTO Media (id, title, type, onchapter, chapters, status)
     VALUES (:id, :title, :type, :onchapter, :chapters, :status);', 
    values
)

https://stackoverflow.com/a/69460935

Added a more complete list of fields that may come out of `aprslib.parse()` to the database schema. How to insert the results of that function into the database without having all the columns represented each time? Seems like we need to keep around a list of columns, drop it into the statement, but also add null dict items to the dict that comes out of `aprslib.parse()`. https://stackoverflow.com/a/16698310 ``` values = { 'title':'jack', 'type':None, 'genre':'Action', 'onchapter':None,'chapters':6,'status':'Ongoing' } cur.execute( 'INSERT INTO Media (id, title, type, onchapter, chapters, status) VALUES (:id, :title, :type, :onchapter, :chapters, :status);', values ) ``` https://stackoverflow.com/a/69460935
mattbk added 1 commit 2023-04-14 22:06:52 -05:00
Store frames in database, remove frames older than chosen age. 63963c0ade
W1CDN commented 2023-04-14 22:07:10 -05:00
Author
Owner
Copy Link
Copy Source

This is looking pretty good...

This is looking pretty good...
W1CDN commented 2023-04-15 11:23:55 -05:00
Author
Owner
Copy Link
Copy Source

Copied new code over to kiss_and_db.py and it seems to run.

  • pull packets from db instead of files
Copied new code over to `kiss_and_db.py` and it seems to run. - [x] pull packets from db instead of files
W1CDN commented 2023-04-15 13:22:08 -05:00
Author
Owner
Copy Link
Copy Source

Hmmm...
image

Hmmm... ![image](/attachments/2d47ede5-5963-45c7-b5c3-b3c01379ffa7)
image.png
673 KiB
W1CDN commented 2023-04-15 13:24:14 -05:00
Author
Owner
Copy Link
Copy Source

I think it was the DB Browser--I made a manual edit to the structure and didn't save it.

I think it was the DB Browser--I made a manual edit to the structure and didn't save it.
mattbk added 1 commit 2023-04-15 13:27:09 -05:00
Read from database for API. 6e6693bfbc
W1CDN commented 2023-04-15 13:40:53 -05:00
Author
Owner
Copy Link
Copy Source
  • fix this issue
$ python kiss_and_db.py 
  File "kiss_and_db.py", line 93
    attrib_names = ', '.join(f'"{w}"' for w in a.keys())
    
File "kiss_and_db.py", line 96
    sql = f"INSERT INTO frames ({attrib_names}) VALUES ({attrib_values})"

On production ^, Python 3.7.3.

But it works on dev machine, Python 3.9.2.

Currently will not run correctly at https://digi.w1cdn.net.

- [x] fix this issue ``` $ python kiss_and_db.py File "kiss_and_db.py", line 93 attrib_names = ', '.join(f'"{w}"' for w in a.keys()) File "kiss_and_db.py", line 96 sql = f"INSERT INTO frames ({attrib_names}) VALUES ({attrib_values})" ``` On production ^, Python 3.7.3. But it works on dev machine, Python 3.9.2. Currently will not run correctly at https://digi.w1cdn.net.
mattbk added 1 commit 2023-04-15 14:20:10 -05:00
Try to get working on production. 14effc9746
W1CDN commented 2023-04-15 14:22:34 -05:00
Author
Owner
Copy Link
Copy Source

The above issues fixed with with code changes, but things aren't working using waitress on the production machine. Flask dev server works fine.

Getting 502 Bad Gateway with waitress.

Subprocess call to kiss_and_db.py does not seem to be the cause.

The above issues fixed with with code changes, but things aren't working using waitress on the production machine. Flask dev server works fine. Getting `502 Bad Gateway` with waitress. Subprocess call to `kiss_and_db.py` does not seem to be the cause.
mattbk added 1 commit 2023-04-15 16:37:35 -05:00
Update waitress IP to make it work on production. 7453d50598
W1CDN commented 2023-04-16 09:55:03 -05:00
Author
Owner
Copy Link
Copy Source

I thought I had the db entry set up right, but running overnight didn't yield new entries...

screen -r back in tells me why:
image

I thought I had the db entry set up right, but running overnight didn't yield new entries... `screen -r` back in tells me why: ![image](/attachments/a11a332c-1ef9-40bb-b2b2-9aad82153ace)
image.png
278 KiB
mattbk added 1 commit 2023-04-16 10:04:54 -05:00
Add missing fields and try to catch sqlite errors. ef5765e87f
W1CDN commented 2023-04-16 10:39:06 -05:00
Author
Owner
Copy Link
Copy Source

Drop this stuff into config.ini:

MYCALL = os.environ.get("MYCALL", "W1CDN")
KISS_HOST = os.environ.get("KISS_HOST", "192.168.0.30")
KISS_PORT = os.environ.get("KISS_PORT", "8001")
Drop this stuff into `config.ini`: ``` MYCALL = os.environ.get("MYCALL", "W1CDN") KISS_HOST = os.environ.get("KISS_HOST", "192.168.0.30") KISS_PORT = os.environ.get("KISS_PORT", "8001") ```
W1CDN commented 2023-04-16 11:32:41 -05:00
Author
Owner
Copy Link
Copy Source

I'll be debugging packets forever, I think!

{'raw': 'FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'FGFFLS', 'to': 'APRS', 'path': ['TCPIP', 'W1CDN-1*'], 'via': '', 'object_name': 'FGFFLS2AA', 'alive': True, 'raw_timestamp': '170815z', 'timestamp': 1681719300, 'format': 'object', 'posambiguity': 0, 'symbol': 'w', 'symbol_table': '\\', 'latitude': 47.885, 'longitude': -96.965, 'comment': 'FLOOD }k0I8Pdd`P:88P{G2AAA', 'object_format': 'uncompressed'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}

{'raw': 'W1CDN-1>APDW16,K0UND-2*:}FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'W1CDN-1', 'to': 'APDW16', 'path': ['K0UND-2*'], 'via': '', 'format': 'thirdparty', 'subpacket': {'raw': 'FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'FGFFLS', 'to': 'APRS', 'path': ['TCPIP', 'W1CDN-1*'], 'via': '', 'object_name': 'FGFFLS2AA', 'alive': True, 'raw_timestamp': '170815z', 'timestamp': 1681719300, 'format': 'object', 'posambiguity': 0, 'symbol': 'w', 'symbol_table': '\\', 'latitude': 47.885, 'longitude': -96.965, 'comment': 'FLOOD }k0I8Pdd`P:88P{G2AAA', 'object_format': 'uncompressed'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}
I'll be debugging packets forever, I think! ``` {'raw': 'FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'FGFFLS', 'to': 'APRS', 'path': ['TCPIP', 'W1CDN-1*'], 'via': '', 'object_name': 'FGFFLS2AA', 'alive': True, 'raw_timestamp': '170815z', 'timestamp': 1681719300, 'format': 'object', 'posambiguity': 0, 'symbol': 'w', 'symbol_table': '\\', 'latitude': 47.885, 'longitude': -96.965, 'comment': 'FLOOD }k0I8Pdd`P:88P{G2AAA', 'object_format': 'uncompressed'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} {'raw': 'W1CDN-1>APDW16,K0UND-2*:}FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'W1CDN-1', 'to': 'APDW16', 'path': ['K0UND-2*'], 'via': '', 'format': 'thirdparty', 'subpacket': {'raw': 'FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'FGFFLS', 'to': 'APRS', 'path': ['TCPIP', 'W1CDN-1*'], 'via': '', 'object_name': 'FGFFLS2AA', 'alive': True, 'raw_timestamp': '170815z', 'timestamp': 1681719300, 'format': 'object', 'posambiguity': 0, 'symbol': 'w', 'symbol_table': '\\', 'latitude': 47.885, 'longitude': -96.965, 'comment': 'FLOOD }k0I8Pdd`P:88P{G2AAA', 'object_format': 'uncompressed'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} ```
mattbk added 1 commit 2023-04-16 16:45:32 -05:00
Add kiss stuff to config. 2ca627f973
W1CDN merged commit 467ec11522 into main 2023-04-16 16:45:54 -05:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
W1CDN (Matt )
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: W1CDN/aprs_tool#20