W1CDN/aprs_tool
0
0
Fork 0
Code Issues 45 Pull Requests Projects Releases Activity

Add background KISS connection, log frames to database #20

Merged
W1CDN merged 10 commits from add-kiss into main 2023-04-16 16:45:54 -05:00
Conversation 15 Commits 10 Files Changed 9 +348 -5
W1CDN commented 2023-04-13 17:19:35 -05:00
Owner
Copy Link

Closes #5.

Closes #5.
W1CDN added 1 commit 2023-04-13 17:19:36 -05:00 
It seems to simultaneously run the API and keep the KISS connection open 
in the background.
W1CDN commented 2023-04-13 20:30:03 -05:00
Author
Owner
Copy Link

https://stackoverflow.com/a/7224186

https://stackoverflow.com/a/7224186
W1CDN commented 2023-04-13 20:46:10 -05:00
Author
Owner
Copy Link

I think I should be using https://github.com/python-aprs/aprs3.

For versions of the KISS transports which do NOT automatically encode/decode APRS data, see kiss3.

I think I should be using https://github.com/python-aprs/aprs3. > For versions of the KISS transports which do NOT automatically encode/decode APRS data, see kiss3.
mattbk added 2 commits 2023-04-13 21:11:08 -05:00
W1CDN commented 2023-04-13 21:12:20 -05:00
Author
Owner
Copy Link

Just barely working now (packet into database), but I don't understand the structure of the APRSFrame object, so it's going in as a string right now.

> repl(frame)
APRSFrame(destination=Address(callsign=b'APZ001', ssid=0, digi=False), source=Address(callsign=b'W1CDN', ssid=0, digi=False), path=[Address(callsign=b'K0UND', ssid=2, digi=True)], control=Control(v=b'\x03'), pid=b'\xf0', info=StatusReport(raw=b'>Hello World!', data_type=<DataType.STATUS: b'>'>, data=b'Hello World!', comment=b'', status=b'Hello World!', _timestamp=None))

Switching to aprslib, which parses more easily?
https://aprs-python.readthedocs.io/en/stable/examples.html

Just barely working now (packet into database), but I don't understand the structure of the APRSFrame object, so it's going in as a string right now. ``` > repl(frame) APRSFrame(destination=Address(callsign=b'APZ001', ssid=0, digi=False), source=Address(callsign=b'W1CDN', ssid=0, digi=False), path=[Address(callsign=b'K0UND', ssid=2, digi=True)], control=Control(v=b'\x03'), pid=b'\xf0', info=StatusReport(raw=b'>Hello World!', data_type=<DataType.STATUS: b'>'>, data=b'Hello World!', comment=b'', status=b'Hello World!', _timestamp=None)) ``` Switching to aprslib, which parses more easily? https://aprs-python.readthedocs.io/en/stable/examples.html
W1CDN commented 2023-04-13 21:37:08 -05:00
Author
Owner
Copy Link

Next up,

  • move code from tcp_kiss_send_recv.py to kiss_and_db.py.
  • test on SBC to see if there is a ton of load from the subprocess

Now need to know all the possible fields that might come up, so they can be added to the db (or maybe write a function to add new fields on the fly?).

Are fields listed in the spec? I think so, see here: https://aprs-python.readthedocs.io/en/stable/parse_formats.html

image

**Next up**, - [x] move code from `tcp_kiss_send_recv.py` to `kiss_and_db.py`. - [x] test on SBC to see if there is a ton of load from the subprocess Now need to know all the possible fields that might come up, so they can be added to the db (or maybe write a function to add new fields on the fly?). Are fields listed in the spec? I think so, see here: https://aprs-python.readthedocs.io/en/stable/parse_formats.html ![image](/attachments/68d13e62-7b38-4e45-801e-8d83b4077673)
image.png
281 KiB
mattbk added 1 commit 2023-04-13 21:38:55 -05:00
W1CDN commented 2023-04-14 12:24:00 -05:00
Author
Owner
Copy Link

Subprocess was running all night and doesn't seem to have broken anything.

Subprocess was running all night and doesn't seem to have broken anything.
W1CDN commented 2023-04-14 19:55:01 -05:00
Author
Owner
Copy Link

Added a more complete list of fields that may come out of aprslib.parse() to the database schema.

How to insert the results of that function into the database without having all the columns represented each time?

Seems like we need to keep around a list of columns, drop it into the statement, but also add null dict items to the dict that comes out of aprslib.parse().

https://stackoverflow.com/a/16698310

values = {
    'title':'jack', 'type':None, 'genre':'Action', 
    'onchapter':None,'chapters':6,'status':'Ongoing'
}
cur.execute(
    'INSERT INTO Media (id, title, type, onchapter, chapters, status)
     VALUES (:id, :title, :type, :onchapter, :chapters, :status);', 
    values
)

https://stackoverflow.com/a/69460935

Added a more complete list of fields that may come out of `aprslib.parse()` to the database schema. How to insert the results of that function into the database without having all the columns represented each time? Seems like we need to keep around a list of columns, drop it into the statement, but also add null dict items to the dict that comes out of `aprslib.parse()`. https://stackoverflow.com/a/16698310 ``` values = { 'title':'jack', 'type':None, 'genre':'Action', 'onchapter':None,'chapters':6,'status':'Ongoing' } cur.execute( 'INSERT INTO Media (id, title, type, onchapter, chapters, status) VALUES (:id, :title, :type, :onchapter, :chapters, :status);', values ) ``` https://stackoverflow.com/a/69460935
mattbk added 1 commit 2023-04-14 22:06:52 -05:00
W1CDN commented 2023-04-14 22:07:10 -05:00
Author
Owner
Copy Link

This is looking pretty good...

This is looking pretty good...
W1CDN commented 2023-04-15 11:23:55 -05:00
Author
Owner
Copy Link

Copied new code over to kiss_and_db.py and it seems to run.

  • pull packets from db instead of files
Copied new code over to `kiss_and_db.py` and it seems to run. - [x] pull packets from db instead of files
W1CDN commented 2023-04-15 13:22:08 -05:00
Author
Owner
Copy Link

Hmmm...
image

Hmmm... ![image](/attachments/2d47ede5-5963-45c7-b5c3-b3c01379ffa7)
image.png
673 KiB
W1CDN commented 2023-04-15 13:24:14 -05:00
Author
Owner
Copy Link

I think it was the DB Browser--I made a manual edit to the structure and didn't save it.

I think it was the DB Browser--I made a manual edit to the structure and didn't save it.
mattbk added 1 commit 2023-04-15 13:27:09 -05:00
W1CDN commented 2023-04-15 13:40:53 -05:00
Author
Owner
Copy Link
  • fix this issue
$ python kiss_and_db.py 
  File "kiss_and_db.py", line 93
    attrib_names = ', '.join(f'"{w}"' for w in a.keys())
    
File "kiss_and_db.py", line 96
    sql = f"INSERT INTO frames ({attrib_names}) VALUES ({attrib_values})"

On production ^, Python 3.7.3.

But it works on dev machine, Python 3.9.2.

Currently will not run correctly at https://digi.w1cdn.net.

- [x] fix this issue ``` $ python kiss_and_db.py File "kiss_and_db.py", line 93 attrib_names = ', '.join(f'"{w}"' for w in a.keys()) File "kiss_and_db.py", line 96 sql = f"INSERT INTO frames ({attrib_names}) VALUES ({attrib_values})" ``` On production ^, Python 3.7.3. But it works on dev machine, Python 3.9.2. Currently will not run correctly at https://digi.w1cdn.net.
mattbk added 1 commit 2023-04-15 14:20:10 -05:00
W1CDN commented 2023-04-15 14:22:34 -05:00
Author
Owner
Copy Link

The above issues fixed with with code changes, but things aren't working using waitress on the production machine. Flask dev server works fine.

Getting 502 Bad Gateway with waitress.

Subprocess call to kiss_and_db.py does not seem to be the cause.

The above issues fixed with with code changes, but things aren't working using waitress on the production machine. Flask dev server works fine. Getting `502 Bad Gateway` with waitress. Subprocess call to `kiss_and_db.py` does not seem to be the cause.
mattbk added 1 commit 2023-04-15 16:37:35 -05:00
W1CDN commented 2023-04-16 09:55:03 -05:00
Author
Owner
Copy Link

I thought I had the db entry set up right, but running overnight didn't yield new entries...

screen -r back in tells me why:
image

I thought I had the db entry set up right, but running overnight didn't yield new entries... `screen -r` back in tells me why: ![image](/attachments/a11a332c-1ef9-40bb-b2b2-9aad82153ace)
image.png
278 KiB
mattbk added 1 commit 2023-04-16 10:04:54 -05:00
W1CDN commented 2023-04-16 10:39:06 -05:00
Author
Owner
Copy Link

Drop this stuff into config.ini:

MYCALL = os.environ.get("MYCALL", "W1CDN")
KISS_HOST = os.environ.get("KISS_HOST", "192.168.0.30")
KISS_PORT = os.environ.get("KISS_PORT", "8001")
Drop this stuff into `config.ini`: ``` MYCALL = os.environ.get("MYCALL", "W1CDN") KISS_HOST = os.environ.get("KISS_HOST", "192.168.0.30") KISS_PORT = os.environ.get("KISS_PORT", "8001") ```
W1CDN commented 2023-04-16 11:32:41 -05:00
Author
Owner
Copy Link

I'll be debugging packets forever, I think!

{'raw': 'FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'FGFFLS', 'to': 'APRS', 'path': ['TCPIP', 'W1CDN-1*'], 'via': '', 'object_name': 'FGFFLS2AA', 'alive': True, 'raw_timestamp': '170815z', 'timestamp': 1681719300, 'format': 'object', 'posambiguity': 0, 'symbol': 'w', 'symbol_table': '\\', 'latitude': 47.885, 'longitude': -96.965, 'comment': 'FLOOD }k0I8Pdd`P:88P{G2AAA', 'object_format': 'uncompressed'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}

{'raw': 'W1CDN-1>APDW16,K0UND-2*:}FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'W1CDN-1', 'to': 'APDW16', 'path': ['K0UND-2*'], 'via': '', 'format': 'thirdparty', 'subpacket': {'raw': 'FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'FGFFLS', 'to': 'APRS', 'path': ['TCPIP', 'W1CDN-1*'], 'via': '', 'object_name': 'FGFFLS2AA', 'alive': True, 'raw_timestamp': '170815z', 'timestamp': 1681719300, 'format': 'object', 'posambiguity': 0, 'symbol': 'w', 'symbol_table': '\\', 'latitude': 47.885, 'longitude': -96.965, 'comment': 'FLOOD }k0I8Pdd`P:88P{G2AAA', 'object_format': 'uncompressed'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'}
I'll be debugging packets forever, I think! ``` {'raw': 'FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'FGFFLS', 'to': 'APRS', 'path': ['TCPIP', 'W1CDN-1*'], 'via': '', 'object_name': 'FGFFLS2AA', 'alive': True, 'raw_timestamp': '170815z', 'timestamp': 1681719300, 'format': 'object', 'posambiguity': 0, 'symbol': 'w', 'symbol_table': '\\', 'latitude': 47.885, 'longitude': -96.965, 'comment': 'FLOOD }k0I8Pdd`P:88P{G2AAA', 'object_format': 'uncompressed'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} {'raw': 'W1CDN-1>APDW16,K0UND-2*:}FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'W1CDN-1', 'to': 'APDW16', 'path': ['K0UND-2*'], 'via': '', 'format': 'thirdparty', 'subpacket': {'raw': 'FGFFLS>APRS,TCPIP,W1CDN-1*:;FGFFLS2AA*170815z4753.10N\\09657.90Ww FLOOD }k0I8Pdd`P:88P{G2AAA', 'from': 'FGFFLS', 'to': 'APRS', 'path': ['TCPIP', 'W1CDN-1*'], 'via': '', 'object_name': 'FGFFLS2AA', 'alive': True, 'raw_timestamp': '170815z', 'timestamp': 1681719300, 'format': 'object', 'posambiguity': 0, 'symbol': 'w', 'symbol_table': '\\', 'latitude': 47.885, 'longitude': -96.965, 'comment': 'FLOOD }k0I8Pdd`P:88P{G2AAA', 'object_format': 'uncompressed'}, 'station_call': 'W1CDN-1', 'station_lat': '47.941500', 'station_lon': '-97.027000'} ```
mattbk added 1 commit 2023-04-16 16:45:32 -05:00
W1CDN merged commit 467ec11522 into main 2023-04-16 16:45:54 -05:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: W1CDN/aprs_tool#20
No description provided.
Delete Branch "add-kiss"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?