Fix test framework running against real devices #1235

sdb9696 · 2024-11-08T15:08:16Z

No description provided.

codecov · 2024-11-08T15:10:28Z

Codecov Report

Attention: Patch coverage is 76.92308% with 3 lines in your changes missing coverage. Please review.

Project coverage is 92.57%. Comparing base (32671da) to head (420f428).
Report is 193 commits behind head on master.

Files with missing lines	Patch %	Lines
kasa/smart/smartdevice.py	66.66%	1 Missing and 1 partial ⚠️
kasa/iot/iotdevice.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1235      +/-   ##
==========================================
- Coverage   92.60%   92.57%   -0.04%     
==========================================
  Files         101      101              
  Lines        6644     6657      +13     
  Branches      706      707       +1     
==========================================
+ Hits         6153     6163      +10     
- Misses        370      372       +2     
- Partials      121      122       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rytilahti

Tested on L530 and it did seemingly work (i.e., I saw the state of the bulb changing), but got some errors from a completely unrelated model:

FAILED kasa/tests/test_bulb.py::test_bulb_sysinfo[L510B(EU)_3.0_1.0.5.json-SMART] - voluptuous.error.MultipleInvalid: expected float for dictionary value @ data['longitude']
FAILED kasa/tests/test_bulb.py::test_hsv_on_non_color[L510B(EU)_3.0_1.0.5.json-SMART] - assert not True
FAILED kasa/tests/test_bulb.py::test_non_variable_temp[L510B(EU)_3.0_1.0.5.json-SMART] - Failed: DID NOT RAISE <class 'kasa.exceptions.KasaException'>

kasa/tests/device_fixtures.py

ryenitcher · 2024-11-09T01:13:01Z

I can confirm that the tests now no longer crash for the EP40M, however the following errors remain:

======================================================================================== short test summary info ========================================================================================= 
FAILED tests\test_childdevice.py::test_child_time[P300(EU)_1.0_1.0.13.json-SMART] - AssertionError: assert FakeDatetime(2024, 11, 8, 17, 56, 32, tzinfo=zoneinfo.ZoneInfo(key='America/Denver')) != FakeDatetime(2024, 11, 8, 17, 56, 32, tzinfo=tzlocal())
FAILED tests\test_readme_examples.py::test_tutorial_examples - ValueError: Cannot find module=docs/tutorial.py

Results (103.99s (0:01:43)):
     458 passed
       2 failed
         - kasa\tests/test_childdevice.py:127 test_child_time[P300(EU)_1.0_1.0.13.json-SMART]
         - kasa\tests/test_readme_examples.py:140 test_tutorial_examples
    13290 skipped

I believe that the docs failure is due to the test being run from the within ./kasa directory, not the top level directory where the docs folder is. The python path would need updated to include the top level directory in order for this to run properly from the ./kasa or ./kasa/tests directory. (I forgot to activate the venv before running pytest, but the test fails even with the venv activated)

Meanwhile it appears that the time test may be failing because the fallback time is the same as the current time, which may be the same as the actual device time. I suggest using the unix epoch instead as the fallback time. Running the tests multiple times in a row shows that it is an intermittent failure, meaning it depends on if you get lucky and how bad your network latency is.

======================================================================================== short test summary info ========================================================================================= 
FAILED tests\test_readme_examples.py::test_tutorial_examples - ValueError: Cannot find module=docs/tutorial.py

Results (109.10s (0:01:49)):
     459 passed
       1 failed
         - kasa\tests/test_readme_examples.py:140 test_tutorial_examples
    13290 skipped

sdb9696 · 2024-11-11T15:28:11Z

Tested on L530 and it did seemingly work (i.e., I saw the state of the bulb changing), but got some errors from a completely unrelated model:

This should now be fixed

I believe that the docs failure is due to the test being run from the within ./kasa directory, not the top level directory where the docs folder is.

That's correct. You should now be able to run this with uv run pytest --ip=<ip address> --username=<username> --password=<password> from the root project directory. Remember to include the = between option and value. Works ok for me on windows.

Meanwhile it appears that the time test may be failing because the fallback time is the same as the current time, which may be the same as the actual device time.

I've now excluded this from physical device testing so it should be skipped.

rytilahti · 2024-11-11T16:07:33Z

A couple of errors, but the model is now correct. Copying a couple of lines from the source for context:

FAILED tests/smart/modules/test_lighttransition.py::test_module_v1[L530E(EU)_3.0_1.0.6.json-SMART] - AssertionError: assert 'smooth_transitions' in {'smooth_transition_off': Smooth transition off (smooth_transition_off): 0 (range: 0-60), 'smooth_transition_on': Smooth transition on (smooth_transition_on): 0 (range...
>       assert "smooth_transitions" in light_transition._module_features
E       AssertionError: assert 'smooth_transitions' in {'smooth_transition_off': Smooth transition off (smooth_transition_off): 0 (range: 0-60), 'smooth_transition_on': Smooth transition on (smooth_transition_on): 0 (range: 0-60)}
E        +  where {'smooth_transition_off': Smooth transition off (smooth_transition_off): 0 (range: 0-60), 'smooth_transition_on': Smooth transition on (smooth_transition_on): 0 (range: 0-60)} = <Module LightTransition (on_off_gradually) for 

FAILED tests/test_common_modules.py::test_light_effect_brightness[L530E(EU)_3.0_1.0.6.json-SMART] - assert 1 == 50
        await light_module.set_brightness(50)
        await dev.update()
        assert light_effect.effect == light_effect.LIGHT_EFFECTS_OFF
>       assert light_module.brightness == 50
E       assert 1 == 50
E        +  where 1 = <Module Light (light) for 192.168.xx.xx>.brightness

kasa/device.py

tests/test_device_factory.py

ryenitcher · 2024-11-14T03:34:06Z

I'm a bit late on the follow up, but for the sake of the records, I can confirm that all device tests now pass when run against the EP40M for the final merged commit 71ae06f (including when run the top level directory).

Results (78.55s (0:01:18)):
     459 passed
    14331 skipped

sdb9696 · 2024-11-14T09:40:47Z

Many thanks @ryenitcher!

Fix test framework against real devices

1fec5cd

sdb9696 mentioned this pull request Nov 8, 2024

Add Additional Firmware Test Fixures #1234

Merged

rytilahti reviewed Nov 9, 2024

View reviewed changes

kasa/tests/device_fixtures.py Outdated Show resolved Hide resolved

rytilahti mentioned this pull request Nov 9, 2024

Fixup contributing.md for running test against a real device #1236

Merged

sdb9696 added 5 commits November 11, 2024 08:48

Merge branch 'master' into janitor/fix_test_ip

be59601

Merge remote-tracking branch 'upstream/master' into janitor/fix_test_ip

a12c495

Fix bulb tests

42a8980

Enable getting username and password from environment vars

81cd8aa

Exclude test_child_time from physical device tests

291121d

sdb9696 requested a review from rytilahti November 11, 2024 15:28

rytilahti reviewed Nov 11, 2024

View reviewed changes

kasa/device.py Show resolved Hide resolved

tests/test_device_factory.py Show resolved Hide resolved

sdb9696 added 3 commits November 11, 2024 16:52

Revert time test back to original time

05bac8c

Fix tests

7c305a4

Make model-region private

3c168a9

sdb9696 requested a review from rytilahti November 11, 2024 17:22

sdb9696 added the maintenance Project improvements and maintenance label Nov 11, 2024

rytilahti approved these changes Nov 11, 2024

View reviewed changes

Add reasons to requires_dummy marks

420f428

sdb9696 added this to the 0.8.0 milestone Nov 11, 2024

sdb9696 merged commit 71ae06f into master Nov 11, 2024
26 of 28 checks passed

sdb9696 deleted the janitor/fix_test_ip branch November 11, 2024 17:41

sdb9696 mentioned this pull request Nov 14, 2024

Add smartcamera devices to supported docs #1257

Merged

sdb9696 mentioned this pull request Nov 25, 2024

Prepare 0.8.0 #1312

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix test framework running against real devices #1235

Fix test framework running against real devices #1235

Uh oh!

sdb9696 commented Nov 8, 2024

Uh oh!

codecov bot commented Nov 8, 2024 •

edited

Loading

Uh oh!

rytilahti left a comment

Uh oh!

Uh oh!

ryenitcher commented Nov 9, 2024 •

edited

Loading

Uh oh!

sdb9696 commented Nov 11, 2024

Uh oh!

rytilahti commented Nov 11, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryenitcher commented Nov 14, 2024

Uh oh!

sdb9696 commented Nov 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Fix test framework running against real devices #1235

Fix test framework running against real devices #1235

Uh oh!

Conversation

sdb9696 commented Nov 8, 2024

Uh oh!

codecov bot commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rytilahti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryenitcher commented Nov 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdb9696 commented Nov 11, 2024

Uh oh!

rytilahti commented Nov 11, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryenitcher commented Nov 14, 2024

Uh oh!

sdb9696 commented Nov 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Nov 8, 2024 •

edited

Loading

ryenitcher commented Nov 9, 2024 •

edited

Loading