[BUG] Full files not being transfered when using FTP or WebDAV

[y2kdread]

Describe the bug

I noticed when doing transfers of files over WebDAV and FTP that the full files are not getting copied over. I tested using different speed factors, and it didn't make a difference:

FTP using Filezilla
WebDAV using Windows Native Client
WebDAV using NetDrive3 (did not complete, thew an error)
WebDAV using RClone (did not complete, threw an error)

To Reproduce

1. Start transfering files using either WebDAV or FTP client
2. Compare the file sizes of FTP/WebDAV to the file contents on the SD card

Expected behavior

Full files should be transferred over.

Screenshots

Files from the SD (not sure why there are few files as compared to WebDAV/FTP

Files when checking sizes on the WebDAV mount:

Files after doing WebDAV transfer

Files after doing FTP transfer

Error from RClone

Error from NetDrive WebDAV

ESP3D Firmware:

• ESP3D FW version: 3.0.0.a225 (pulled from gethub on 7-NOV-2023)
• ESP3D-WebUI Version: 3.0.0-a42.Me2
• Wifi mode: Client Mode
• Flash method: PlatformIO
• Any change done in FW: None

Target Firmware:

Board used (please complete the following information):

• MCU: ESP32
• Name: Fysetc SD-Wifi-Pro
• Flash size: 8M

Additional context

If it matters, I've attached a csv that has the folder structure and file names.
FILES.csv

[luc-github]

I have used Filezilla pro as FTP and WebDav client
and did not get any issue - be sure your client allow only one connection at once
No error on FTP for 250 files so far

No error on webdav client set to use one connection at a time

I know microsoft webdav is trying to use multiple connections so I only use it for atomical copy not batch

I do not know RCclone neither NetDrive but they may have also a setting to limit same as fillezilla

the FILES.csv is empty so I cannot check it but here some things to check:
1 - check your ESP signal strength if too low it may loose some packet
2 - be sure there are only one client at once opened, if you have a drive mounted as webdav and you want to use ftp, unmount webdav first as both could do request in same time and it will failed on ESP web server for sure because it a really a light http server

Also I am not clear you issue is upload on SD or download from SD ?
My test are upload to SD ~250 files each time, my github sources files for ESP3d and ESP3D-TFT

[y2kdread]

Would you mind sharing your configuration files? I can't figure out what's causing this and want to make sure I am not missing anything

[luc-github]

configuration.h.txt
I doubt issue is in configuration.h I changed the minimum and use it as Client

Should should check the points I mentionned above: signal strengh, simultaneous connections

[y2kdread]

It looked like my configured was pretty similar to yours, so I don't think that was an issue.

I did more testing and can't get the transfers to work correctly. I figured the most direct way to do the transfer was using Filezilla making sure that it is only using 1 connection to the SD card. On FTP, with nothing else connected to the SD card anything larger then 1 kb doesn't transfer at all. I've attached my FTP log to this comment.

ftp.log

[y2kdread]

Also -

I confirmed that the wifi connection is good, I am testing it in the same room as one of my access points and it is getting a -34 dBm signal. I also updated the card with your most recent changes to the 3.0 firmware.

[luc-github]

@y2kdread how can you get dBm? webui/Fw only give % of signal strength
Check what [ESP420] give you

[y2kdread]

I got that measurement from my router. The SD card shows the WiFi signal at 100%

[luc-github]

I do not understand how your router can see what level of signal ESP3D see, sorry

can you share your complete [ESP420] output please?

[y2kdread]

Here you go.

[ESP420]
chip id: 8438
CPU Freq: 240Mhz
CPU Temp: 68.9C
free mem: 142.14 KB
SDK: v4.4.3
flash size: 8.00 MB
size for update: 3.19 MB
FS type: LittleFS
FS usage: 108.00 KB/1.50 MB
baud: 115200
sleep mode: none
wifi: ON
hostname: esp3d
HTTP port: 80
WebDav port: 8181
Ftp ports: 21,20,55600
sta: ON
mac: 30:83:98:D2:F6:20
SSID: slacker
signal: 100%
phy mode: 11n
channel: 11
ip mode: dhcp
ip: 192.168.5.8
gw: 192.168.5.1
msk: 255.255.255.0
DNS: 192.168.5.1
ap: OFF
mac: 30:83:98:D2:F6:21
i-time: OFF
serial: ON
sd: shared (SD native)
SD updater: ON
Target Fw: unknown
FW ver: 3.0.0.a225
FW arch: ESP32

[luc-github]

this is not normal : SDK: v4.4.3
Should be 4.4.4
What version of esp32 core are you using ?

[y2kdread]

I am following your installation steps, I haven't changed the core version as far as I know

[luc-github]

you are using arduino or platformio ?

[luc-github]

my repository use
[env:esp32dev]
platform = espressif32@6.2.0 https://github.com/luc-github/ESP3D/blob/3.0/platformio.ini#L20
https://github.com/platformio/platform-espressif32/releases/tag/v6.2.0
Which is based on 2.0.8 which is based on SDK 4.4.4
https://github.com/espressif/arduino-esp32/releases/tag/2.0.8

So I am wondering why you have older core

[y2kdread]

The platform was incorrect for me, I generated it from the configurator, and it was set as:

[env:esp32_8MB_wifi]
platform = espressif32@6.0.1

I updated the setting and regenerated the firmware; I updated it using the WebUI and it reflected the new version from the [ESP420] command. I saw the same behavior with downloads over Filezilla where files over 1 kb weren't being transferred correctly.

Next, I erased and flashed SD card using VSCode. I then uploaded the full WebUI and configured the dashboard. By mistake, I left the daughterboard in flash mode (switch 1 off, switch 2 on) and tested. With the daughterboard in this mode everything transferred correctly.

After this, I tested with the daughterboard in SD reader mode (switch 1 off, switch 2 off) and saw the same behavior where anything over 1 kb didn't transfer correctly. I also tested in my CPAP machine and that also had the issue with file transfers. I also tried downloading a file over 1 kb through the WebUI and received a connection timed out error.

As far as I can tell, nothing else is using the SD card. I even went as far as to disable WebDAV while doing the testing.

I've attached my various configuration files and my list of files and directories.

files.csv
platformio.ini.txt
preferences.json
configuration.h.txt

[luc-github]

ok per your description you have no issue when using on daugther board switch 1 on switch 2 off
but having issue out of daughter board.
so issue seems related to switch multiplexer of the sd

does the issue happen if you only transfert 1 file?
or it randomly happen on batch of files

[y2kdread]

This does happen every time when trying to transfer a single file over FTP or using the WebUI.

[luc-github]

I need to setup a system to reproduce your issue because all my readers are micro sd

and for some reason using sd reader on my pc does not power enough the sd so cannot use it

[y2kdread]

sounds good - I look forward to seeing what you find.

[luc-github]

I found a reader that seems ok with my PC

• I tested with original FW and microsoft windows see the SD mounted but the Web Page cannot see the SD
  That is because it is checking the CS sense pin, and Windows keep this pin enable so it prevent to do the switch

Doing an eject does not solve the issue because then it stop to power the SD card - so no more response from ESP32 web server

Current ESP3D version does not handle the CS sense and that disturb the multiplexer that make switching not reliable

I did a setup with a SD reader only

ESP3D work as expected it switch to SD when needed and release SD once done - I will add the CS sense check later to improve reliability in case of usage on PC Reader

I have transfered by webdav using Filezilla Pro 3170 files from PC to SD and no issue - it just took some time

The file sizes are from Several MB to few bytes:

Now I am doing download from SD to PC using WebDav and FileZilla Pro of the SD content which is now 3173
One folder contain 1000 files so initial recursive transfer is blocked during time the App is doing the files listing to prepare the transfer but it resume after as expected

At the end I got full tranfer, no failure and correct numbers of files

So sorry but I cannot duplicate your issue - I can only guess the app you use does not allow to use one session at a time

[luc-github]

closing issue then

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.