Handle breaking errors in kiss_and_db.py #38

Merged

W1CDN merged 9 commits from fix-kiss into main 2023-12-13 20:20:59 -06:00

Conversation 15 Commits 9 Files Changed 4

+101 -59

W1CDN commented 2023-08-26 19:26:28 -05:00

Owner

Copy link

Copy source

Closes #37.

Started with an example mixed together from https://github.com/python-aprs/aprs3 and https://github.com/python-aprs/kiss3/blob/main/examples/tcp_async.py. The await was missing from the first example, so beware.

Closes #37. Started with an example mixed together from https://github.com/python-aprs/aprs3 and https://github.com/python-aprs/kiss3/blob/main/examples/tcp_async.py. The `await` was missing from the first example, so beware.

W1CDN added the bug label 2023-08-26 19:26:28 -05:00

W1CDN self-assigned this 2023-08-26 19:26:28 -05:00

W1CDN added 1 commit 2023-08-26 19:26:30 -05:00

Stub out fix-kiss. e1211fe108

W1CDN commented 2023-08-26 19:35:20 -05:00

Author

Owner

Copy link

Copy source

Doesn't seem like test_async.py is picking up anything from dw, even though dw reports the connection.

Doesn't seem like `test_async.py` is picking up anything from dw, even though dw reports the connection.

W1CDN added 1 commit 2023-12-10 13:44:57 -06:00

Commit deployed changes from a while back. dc7d4ed8a3

W1CDN commented 2023-12-10 13:59:33 -06:00

Author

Owner

Copy link

Copy source

Got local environment going again and maybe I'll have better luck debugging this. May create another branch to try readme example at https://github.com/python-aprs/aprs3.

Got local environment going again and maybe I'll have better luck debugging this. May create another branch to try readme example at https://github.com/python-aprs/aprs3.

W1CDN commented 2023-12-10 17:08:31 -06:00

Author

Owner

Copy link

Copy source

Well there's your problem:
https://owenduffy.net/blog/?p=2969

Status

The host cannot ascertain the TNC status, there are no KISS commands that have a response, much less any that provide any known information.

Equally, the TNC cannot ascertain host status.

Well there's your problem: https://owenduffy.net/blog/?p=2969 ``` Status The host cannot ascertain the TNC status, there are no KISS commands that have a response, much less any that provide any known information. Equally, the TNC cannot ascertain host status. ```

W1CDN commented 2023-12-10 19:45:15 -06:00

Author

Owner

Copy link

Copy source

Wes asked about keepalive on the TCP connection, and I can't see anything like that in the aprs3/kiss3 code, but that doesn't mean it's not another level down from where I'm looking.

But I'm wondering if I could send an empty packet (or one with no path) every x minutes and have that keep the connection alive?

[Wes asked](https://mastodon.radio/@w8wjb@mastodon.hams.social/111558683524663931) about keepalive on the TCP connection, and I can't see anything like that in the aprs3/kiss3 code, but that doesn't mean it's not another level down from where I'm looking. But I'm wondering if I could send an empty packet (or one with no path) every x minutes and have that keep the connection alive?

W1CDN added 1 commit 2023-12-10 21:04:38 -06:00

Flail around. 307c1fcd86

matt commented 2023-12-11 11:32:45 -06:00

First-time contributor

Copy link

Copy source

/home/pi/.local/lib/python3.7/site-packages/kiss/classes.py

class TCPKISS(KISS):
    """KISS TCP Class."""

    def __init__(self, host: str, port: int, strip_df_start: bool = False):
        super().__init__(strip_df_start)
        self.address = (host, int(port))

    def stop(self) -> None:
        self.protocol.transport.close()

    def start(self, **kwargs: Any) -> None:
        """
        Initializes the KISS device and commits configuration.
        """
        _, self.protocol = self.loop.run_until_complete(
            kiss.create_tcp_connection(
                *self.address,
                protocol_kwargs={"decoder": self.decoder},
            ),
        )
        self.loop.run_until_complete(self.protocol.connection_future)
        self._write_defaults(**kwargs)

 ` /home/pi/.local/lib/python3.7/site-packages/kiss/classes.py` ``` class TCPKISS(KISS): """KISS TCP Class.""" def __init__(self, host: str, port: int, strip_df_start: bool = False): super().__init__(strip_df_start) self.address = (host, int(port)) def stop(self) -> None: self.protocol.transport.close() def start(self, **kwargs: Any) -> None: """ Initializes the KISS device and commits configuration. """ _, self.protocol = self.loop.run_until_complete( kiss.create_tcp_connection( *self.address, protocol_kwargs={"decoder": self.decoder}, ), ) self.loop.run_until_complete(self.protocol.connection_future) self._write_defaults(**kwargs) ```

image.png

73 KiB

W1CDN commented 2023-12-11 12:13:07 -06:00

Author

Owner

Copy link

Copy source

The older version of kiss does not use asyncio to handle the TCP connection: https://github.com/ampledata/kiss/blob/5c8baba2ceaa97a57fd93cab24a502011fd39e85/kiss/classes.py#L205

I don't know if switching back to this will fix the connection or break other things.

If this import socket refers to this, I think keepalive is included? https://docs.python.org/3/library/socket.html

But I can't seem to pip3 install the older kiss and aprs packages.

The older version of `kiss` does not use `asyncio` to handle the TCP connection: https://github.com/ampledata/kiss/blob/5c8baba2ceaa97a57fd93cab24a502011fd39e85/kiss/classes.py#L205 I don't know if switching back to this will fix the connection or break other things. If this `import socket` refers to this, I think keepalive is included? https://docs.python.org/3/library/socket.html But I can't seem to `pip3 install` the older `kiss` and `aprs` packages.

matt commented 2023-12-11 14:54:19 -06:00

First-time contributor

Copy link

Copy source

Started 11:50, seems to have quit/hung ~14:27 CST.
Seems hung after this. Packets came in after this, but nothing in log.
Is it telemetry that is causing issues?
This is where I am using aprslib instead of aprs3 for parsing into tabular format.

Started 11:50, seems to have quit/hung ~14:27 CST. Seems hung after this. Packets came in after this, but nothing in log. Is it telemetry that is causing issues? This is where I am using `aprslib` instead of `aprs3` for parsing into tabular format.  

image.png

51 KiB

image.png

84 KiB

matt commented 2023-12-11 15:30:08 -06:00

First-time contributor

Copy link

Copy source

Was fooling around with packages earlier and ran into this again. Had to reinstall stuff in right order, but first delete some things.

>~/.local/lib/python3.7/site-packages $ rm -R kiss-7.0.0.dist-info/
>~/.local/lib/python3.7/site-packages $ rm -R kiss3-8.0.0.dist-info/
>~/.local/lib/python3.7/site-packages $ rm -R aprs3-8.0.0.post1.dist-info/
>~/.local/lib/python3.7/site-packages $ pip3 install aprs3 --no-cache-dir

Was fooling around with packages earlier and ran into this again. Had to reinstall stuff in right order, but first delete some things. ``` >~/.local/lib/python3.7/site-packages $ rm -R kiss-7.0.0.dist-info/ >~/.local/lib/python3.7/site-packages $ rm -R kiss3-8.0.0.dist-info/ >~/.local/lib/python3.7/site-packages $ rm -R aprs3-8.0.0.post1.dist-info/ >~/.local/lib/python3.7/site-packages $ pip3 install aprs3 --no-cache-dir ```

W1CDN commented 2023-12-11 21:40:01 -06:00

Author

Owner

Copy link

Copy source

Turned off periodic restart after the messages this afternoon, see also #37.

Turned off periodic restart after the messages this afternoon, see also #37.

matt commented 2023-12-12 09:19:19 -06:00

First-time contributor

Copy link

Copy source

Same thing happening this morning. Breaking in between "Parsed ok" and when I attempt to log the

Also need to think about what characters may need to be escaped for SQLite.

Same thing happening this morning. Breaking in between "Parsed ok" and when I attempt to log the  Also need to think about what characters may need to be escaped for SQLite.

image.png

35 KiB

matt commented 2023-12-12 14:47:36 -06:00

First-time contributor

Copy link

Copy source

Now seeing this?

Parsed ok.
Frame could not be parsed.

Deployed code is a bit ahead what is in the repo right now.

Adding more errors (I think): https://www.freecodecamp.org/news/python-print-exception-how-to-try-except-print-an-error/

Now seeing this? ``` Parsed ok. Frame could not be parsed. ```  Deployed code is a bit ahead what is in the repo right now. Adding more errors (I think): https://www.freecodecamp.org/news/python-print-exception-how-to-try-except-print-an-error/

image.png

319 KiB

matt commented 2023-12-12 15:42:38 -06:00

First-time contributor

Copy link

Copy source

EDIT: Doesn't need to be? https://digi.w1cdn.net/aprs_api/packets?from=N0CBV-2

Example with a single quote? Can it be escaped?
N0CBV-2>APNU3B,KB0UGF-3,WIDE2-1:!4636.05NS09641.52W#PHG7230W2 MNn Wolverton 932'
https://stackoverflow.com/questions/3151146/replace-the-single-quote-character-from-a-string

EDIT: Doesn't need to be? https://digi.w1cdn.net/aprs_api/packets?from=N0CBV-2 Example with a single quote? Can it be escaped? `N0CBV-2>APNU3B,KB0UGF-3,WIDE2-1:!4636.05NS09641.52W#PHG7230W2 MNn Wolverton 932'` https://stackoverflow.com/questions/3151146/replace-the-single-quote-character-from-a-string

matt referenced this pull request 2023-12-12 17:02:07 -06:00

KISS connection seems unstable #37

matt commented 2023-12-12 17:05:13 -06:00

First-time contributor

Copy link

Copy source

Frame could not be parsed. is an overarching log message I added for failure and includes everything from parsing to DB insertion. It's not very helpful.

• this code needs to be restructured, it shouldn't keep trying to put stuff in the database once the packet isn't parsed. Use try except else

https://amiok.net/gitea/W1CDN/aprs_tool/src/branch/fix-kiss/kiss_and_db.py#L75

try:
                try:
                    a = aprslib.parse(str(frame))
                except Exception as error:
                    logging.error("Error with aprslib:", error)
                a['station_call'] = config['Settings']['station_call']
                a['station_lat'] = config['Settings']['station_lat']
                a['station_lon'] = config['Settings']['station_lon']
                a['created_unix'] = int(time.time())

                # Make this a string and deal with it later (probably a mistake)
                a['path'] = str(a['path'])
                if 'subpacket' in a:
                    a['subpacket'] = str(a['subpacket'])
                #logging.debug(a['path'])
                # Store true/false as 1/0
                if 'alive' in a:
                    if a['alive'] == True:
                        a['alive'] = 1
                    else:
                        a['alive'] = 0
                # Build an INSERT statement based on the fields we have from the frame
                attrib_names = ', '.join('"%s"' % w for w in a.keys())
                attrib_values = ", ".join("?" * len(a.keys()))
                logging.debug(attrib_names)

`Frame could not be parsed.` is an overarching log message I added for failure and includes everything from parsing to DB insertion. It's not very helpful. - [x] this code needs to be restructured, it shouldn't keep trying to put stuff in the database once the packet isn't parsed. Use `try except else` https://amiok.net/gitea/W1CDN/aprs_tool/src/branch/fix-kiss/kiss_and_db.py#L75 ``` try: try: a = aprslib.parse(str(frame)) except Exception as error: logging.error("Error with aprslib:", error) a['station_call'] = config['Settings']['station_call'] a['station_lat'] = config['Settings']['station_lat'] a['station_lon'] = config['Settings']['station_lon'] a['created_unix'] = int(time.time()) # Make this a string and deal with it later (probably a mistake) a['path'] = str(a['path']) if 'subpacket' in a: a['subpacket'] = str(a['subpacket']) #logging.debug(a['path']) # Store true/false as 1/0 if 'alive' in a: if a['alive'] == True: a['alive'] = 1 else: a['alive'] = 0 # Build an INSERT statement based on the fields we have from the frame attrib_names = ', '.join('"%s"' % w for w in a.keys()) attrib_values = ", ".join("?" * len(a.keys())) logging.debug(attrib_names) ```

W1CDN added 2 commits 2023-12-12 22:07:33 -06:00

Flail. e7002c712c

Try to catch exceptions. 50085de7db

matt commented 2023-12-13 09:19:35 -06:00

First-time contributor

Copy link

Copy source

Logging errors is not working correctly. These show up in the screen -r output, but not in the log file:

Logging errors is not working correctly. These show up in the `screen -r` output, but not in the log file: 

image.png

137 KiB

matt changed title from Stub out fix-kiss. to Handle breaking errors in kiss_and_db.py 2023-12-13 09:35:56 -06:00

W1CDN added 1 commit 2023-12-13 10:13:33 -06:00

Working snapshot. 26b4081720

W1CDN added 2 commits 2023-12-13 10:24:25 -06:00

Align constants and schema. 79da9de6f7

Add header_raw field. 1894de5d86

W1CDN added 1 commit 2023-12-13 10:40:38 -06:00

Log errors in more detail. 25bffd9b61

W1CDN commented 2023-12-13 10:48:07 -06:00

Author

Owner

Copy link

Copy source

Now that errors are being logged in more detail, I can let this run and see if it keeps working. Then deal with other additions in a different branch.

Now that errors are being logged in more detail, I can let this run and see if it keeps working. Then deal with other additions in a different branch.

W1CDN merged commit 4439ae680e into main 2023-12-13 20:20:59 -06:00

W1CDN deleted branch fix-kiss 2023-12-13 20:20:59 -06:00

W1CDN referenced this issue from a commit 2023-12-13 20:21:01 -06:00

Merge pull request 'Handle breaking errors in kiss_and_db.py' (#38) from fix-kiss into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No labels bug

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

W1CDN (Matt )

No Assignees W1CDN

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: W1CDN/aprs_tool#38