Memory Consumption of UploadPictureAsync

Topics: Developer Forum, User Forum
Oct 6, 2013 at 6:00 AM
Hi,

I am working on a WP8 app that uses a resource intensive task to auto upload a users camera roll photos to Flickr.

A resource intensive task is allowed a max of 11MB as stated here: http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202942%28v=vs.105%29.aspx

The problem I have is that before calling UploadPictureAsync I am currently using 5MB of memory but the moment I call UploadPictureAsync I run out of memory and the background agent crashes.

How can I limit the memory consumption of UploadPictureAsync to allow it to be used in a WP8 background task (i.e. under 11MB)?

Thanks,
Matt
Coordinator
Oct 7, 2013 at 10:57 PM
Hi

I've created a test project, and run the Windows phone analysis tools, and I don't see a spike in memory when I use a Stream to transfer the image.

What size images are we talking? Could you provide any sample code?

Sam
Oct 7, 2013 at 11:19 PM
The images are on the order of 2.5MB. From my testing, before the call to UploadPictureAsync the task is consuming 5MB of memory and the moment it crashed it is at 14MB of memory (limit of 11MB). I will provide the actual code when I get home later today. Thanks for all the help.
Nov 5, 2013 at 5:43 AM
Hi guys.

I think I found why the UploadPictureAsync method eats up so much memory. The problem is in the method CreateUploadData in Flickr.cs. The method takes the image stream as a parameter, then converts it to a byte array, and then copies it into the final byte array which is used in the UploadDataAsync method. I think I found a fix for that issue. Instead of returning one big byte array with everything compacted inside, I return two byte arrays, one for the header of the request and one for the footer. In the UploadDataAsync method, after opening the stream, I first write the header, then the image and then the footer. The only downside is that I removed the progress notification since I wasn't using it anyway. The trouble is that I can't seem to get the current branch working. It throws authentication required on all calls even though I do have tokens and everything. My question is this: can I have the source files of the 3.8 release so I can modify them with these changes?

Right now I'm trying to implement an auto uploader for Flickr on WP and one upload of a 2.5MB file peaks at 18MB(out of 20MB assigned). The second upload I'm trying to run crashes with out of memory.
        private byte[][] CreateUploadData(Stream imageStream, string fileName, Dictionary<string, string> parameters, string boundary)
        {
            var oAuth = parameters.ContainsKey("oauth_consumer_key");

            var keys = new string[parameters.Keys.Count];
            parameters.Keys.CopyTo(keys, 0);
            Array.Sort(keys);

            var hashStringBuilder = new StringBuilder(sharedSecret, 2 * 1024);
            var contentStringBuilder = new StringBuilder();

            foreach (string key in keys)
            {

#if !SILVERLIGHT
                // Silverlight < 5 doesn't support modification of the Authorization header, so all data must be sent in post body.
                if (key.StartsWith("oauth")) continue;
#endif
                hashStringBuilder.Append(key);
                hashStringBuilder.Append(parameters[key]);
                contentStringBuilder.Append("--" + boundary + "\r\n");
                contentStringBuilder.Append("Content-Disposition: form-data; name=\"" + key + "\"\r\n");
                contentStringBuilder.Append("\r\n");
                contentStringBuilder.Append(parameters[key] + "\r\n");
            }

            if (!oAuth)
            {
                contentStringBuilder.Append("--" + boundary + "\r\n");
                contentStringBuilder.Append("Content-Disposition: form-data; name=\"api_sig\"\r\n");
                contentStringBuilder.Append("\r\n");
                contentStringBuilder.Append(UtilityMethods.MD5Hash(hashStringBuilder.ToString()) + "\r\n");
            }

            // Photo
            contentStringBuilder.Append("--" + boundary + "\r\n");
            contentStringBuilder.Append("Content-Disposition: form-data; name=\"photo\"; filename=\"" + Path.GetFileName(fileName) + "\"\r\n");
            contentStringBuilder.Append("Content-Type: image/jpeg\r\n");
            contentStringBuilder.Append("\r\n");

            var encoding = new UTF8Encoding();

            byte[] postContents = encoding.GetBytes(contentStringBuilder.ToString());

            byte[] postFooter = encoding.GetBytes("\r\n--" + boundary + "--\r\n");


            byte[][] ret = new byte[2][];
            ret[0] = postContents;
            ret[1] = postFooter;
            return ret;
        }
private void UploadDataAsync(Stream imageStream, string fileName, Uri uploadUri, Dictionary<string, string> parameters, Action<FlickrResult<string>> callback)
        {
            string boundary = "FLICKR_MIME_" + DateTime.Now.ToString("yyyyMMddhhmmss", System.Globalization.DateTimeFormatInfo.InvariantInfo);

            string authHeader = FlickrResponder.OAuthCalculateAuthHeader(parameters);

            byte[][] dataBuffers = CreateUploadData(imageStream, fileName, parameters, boundary);

            byte[] header = dataBuffers[0];
            byte[] footer = dataBuffers[1];


            HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uploadUri);
            req.Method = "POST";
            req.ContentType = "multipart/form-data; boundary=" + boundary;
            if (!String.IsNullOrEmpty(authHeader))
            {
                req.Headers["Authorization"] = authHeader;
            }

            req.BeginGetRequestStream(
                r =>
                {
                    using (Stream reqStream = req.EndGetRequestStream(r))
                    {
                        int bufferSize = 32 * 1024;

                        int uploadedSoFar = 0;

                        while (uploadedSoFar < header.Length)
                        {
                            reqStream.Write(header, uploadedSoFar, Math.Min(bufferSize, header.Length - uploadedSoFar));
                            uploadedSoFar += bufferSize;
                        }


                        byte[] buffer = new byte[bufferSize];
                        int pointer = 0;
                        int read = 0;
                        while ((read = imageStream.Read(buffer, pointer, bufferSize)) > 0)
                        {
                            pointer += read;
                            reqStream.Write(buffer, 0, read);
                        }

                        uploadedSoFar = 0;

                        while (uploadedSoFar < footer.Length)
                        {
                            reqStream.Write(footer, uploadedSoFar, Math.Min(bufferSize, footer.Length - uploadedSoFar));
                            uploadedSoFar += bufferSize;
                        }

                        reqStream.Close();
                    }

                    req.BeginGetResponse(
                        r2 =>
                        {
                            FlickrResult<string> result = new FlickrResult<string>();

                            try
                            {
                                WebResponse res = req.EndGetResponse(r2);
                                StreamReader sr = new StreamReader(res.GetResponseStream());
                                string responseXml = sr.ReadToEnd();
                                sr.Close();

                                XmlReaderSettings settings = new XmlReaderSettings();
                                settings.IgnoreWhitespace = true;
                                XmlReader reader = XmlReader.Create(new StringReader(responseXml), settings);

                                if (!reader.ReadToDescendant("rsp"))
                                {
                                    throw new XmlException("Unable to find response element 'rsp' in Flickr response");
                                }
                                while (reader.MoveToNextAttribute())
                                {
                                    if (reader.LocalName == "stat" && reader.Value == "fail")
                                        throw ExceptionHandler.CreateResponseException(reader);
                                    continue;
                                }

                                reader.MoveToElement();
                                reader.Read();

                                UnknownResponse t = new UnknownResponse();
                                ((IFlickrParsable)t).Load(reader);
                                result.Result = t.GetElementValue("photoid");
                                result.HasError = false;
                            }
                            catch (Exception ex)
                            {
                                if (ex is WebException)
                                {
                                    OAuthException oauthEx = new OAuthException(ex);
                                    if (String.IsNullOrEmpty(oauthEx.Message))
                                        result.Error = ex;
                                    else
                                        result.Error = oauthEx;
                                }
                                else
                                {
                                    result.Error = ex;
                                }
                            }

                            callback(result);

                        }, 
                        this);
                }, 
                this);

        }
Coordinator
Nov 6, 2013 at 2:37 PM
Yes, I think I will have to do something like that. Might take me a while as I've just had my second little girl, so coding hours are significantly reduced at the moment :)
Nov 6, 2013 at 2:40 PM
The upload worked fine with my modifications but the memory consumption kept growing. I did find the reason though. For background agents that are limited to around 20MB of RAM, you need to set the AllowReadStreamBuffering and AllowWriteStreamBuffering to false in order to avoid the OS keeping unnecessary amounts of buffered data in RAM.
Coordinator
Jan 23, 2014 at 5:32 PM
Hi

I've just checked in an adapted version which now uses streams all the way down. The only reason it would create a copy of a stream is if it is nonseekable, as we need to know the length of the content in order to set the content-length header.

If you could give it a try that would be good.

Sam