Code Blog Topics

Some of us love those electrons just a little too much
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

Ddrak wrote:Worth a 'GO' after each line and possibly ending them with a ';'?
I was going to make those both switches, as well as comments and maybe event comment style. But I can say for sure on SQL Server 2012 the GOs aren't needed and neither are the semicolons. Right now the script generator is only putting 2 GOs in that I know of it needs, while I debate the other ones.

But that brings up an important point. Currently I have a target provider name, and most of what I am doing is only if you target SQL server. For example, if you target SQL Server, you probably don't need the semicolons, but you would for MySQL. So that's easy to do since it's already provider aware. And since it's provider aware, I'm not sure I need the switches since I know what every provider expects.

That's going to be the beauty of this chunk of code: you'll be able to capture a database from provider (A) and restore it to provider (B). Theoretically you could capture an Excel spreadsheet to an Oracle database. Though in actuality every single DB provider is a quirky, hot mess, and there will be lots of provider-specific code as this project evolves. But at least it well be well-structured and encapsulated provider specific code :)
Ddrak
Save a Koala, deport an Australian
Posts: 17516
Joined: Thu Jan 02, 2003 3:00 pm
Location: Straya mate!
Contact:

Re: Code Blog Topics

Post by Ddrak »

That's going to be the beauty of this chunk of code: you'll be able to capture a database from provider (A) and restore it to provider (B).
Yeah. In theory. ;)

In practice, unless you've got an exceptionally trivial database, you've made use of provider-specific stuff even at the table level (say, Identity colums vs Sequences) and it's not even worth talking about views or stored procs. (sigh)

Dd
Image
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

Ddrak wrote:
That's going to be the beauty of this chunk of code: you'll be able to capture a database from provider (A) and restore it to provider (B).
Yeah. In theory. ;)

In practice, unless you've got an exceptionally trivial database, you've made use of provider-specific stuff even at the table level
Yeah, it's not possible to do this stuff at a generic level, though the generic dump has enough info to do what we need it for at work, which is mainly to detect columns that have been changed, removed or renamed in the database without anyone telling me. Until you bust out quantifiable metrics ... nobody is sure if they did anything to the database or not.

An interesting exchange:

Me: Did you guys change the database schema? The production system started blowing up.
Guy: We didn't change anything.
Me: That's funny you should say that, because I have a list of the 247 columns that changed.
Guy: I meant to say, we didn't change anything that would affect you.
Me: The production systems blowing up and the clients getting frustrated affect me.
Guy: It wasn't anything that would affect you, though.
Me: Did you remove column X from table Y?
Guy: Yeah, but it wasn't needed.
Me: ...except by the system in production.

It's interesting to note that until I had the output of my DB compare tool in hand, I couldn't even get the guys to admit that anything had changed in the database. Until I started giving them table and column names, they didn't remember shit.
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

The database reconstructor is starting to drag so I cranked out a little windows utility for this series. It will let you surf the XML snapshot and see all the schema table data. Really simple and powerful. It's just a drop down list of schema tables in the XML file and the grid shows the selected table. Oh, and this post includes a link to the ZIP file project.

Here's the new blog post:

DB Snapshot Viewer Tool

Image
Ddrak
Save a Koala, deport an Australian
Posts: 17516
Joined: Thu Jan 02, 2003 3:00 pm
Location: Straya mate!
Contact:

Re: Code Blog Topics

Post by Ddrak »

I 100% agree on having CM on the database schema (and seed data for that matter).

We build a bunch of html files that get compiled to a .chm from the schema file we generate so people can browse around a hyperlinked version of the schema.

Dd
Image
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

I've got it creating the table structure, keys and constraints perfectly. It recreates our new system's table and key structure exactly. Way more work than I originally figured, though.

Views are next.
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

Next blog post is an encrypted file container. Kind of like an encrypted ZIP. In fact, I may add open source compression to it. Microsoft's crypto libraries are really convenient to use, but I may add some open source encryption on top of it. Microsoft's crypto is rumored to have back doors in it, where at least with something like Blowfish you have the source, for whatever that's worth. I believe there is at least one documented case of open source crypto being deliberately and strategically weakened.

I did something like this for work and it really turned out great. In our case, the encrypted repository mostly just serves as a tamper proof container of insurance document templates we can easily deploy to production servers. And there's an HTTP handler which serves up these documents straight from the repository to the user's browser without touching the hard disk.
Ddrak
Save a Koala, deport an Australian
Posts: 17516
Joined: Thu Jan 02, 2003 3:00 pm
Location: Straya mate!
Contact:

Re: Code Blog Topics

Post by Ddrak »

I believe there is at least one documented case of open source crypto being deliberately and strategically weakened.
You mean like DES/3DES was?

Dd
Image
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

One of the things that interests me about Git is sites like GitHub.
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

^^^^--- Oops posted to wrong thread.

The encrypted file container app is almost done. For now it just uses Microsoft's implementation of AES 256. So far it's pretty slick. It's a SAM file structure and each file has a blob of XML meta data, which is encrypted with a key derived from the user password and also contains a stronger key and init vector specific to each file, which should make it a little more secure because every file in the repository has a separate key and IV.

I've been farting around with some cool stuff like elliptic curve crypto and maybe using a second algorithm and XOR the two (or more) together like some popular packages do. And I like the so called "duress features" of some other products. It's easy to have a duress key that makes the container object show decoy contents. Basically two containers, one a decoy. That maybe a little fancy, though.

Either way, just with AES and a few good practices, this is going to be plenty good for the masses. A lightweight but powerful encrypted container app. I may use this myself. I like to keep an encrypted file with all my logins, servers, URLs, etc. handy.
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

Here's a sample of the meta data it stores encrypted for every file in the repository. It even gives the file a unique ID and also allows for custom tags.

Code: Select all

<root>
  <Name>TestSqlServer.xml</Name>
  <Size>9302795</Size>
  <DateCreated>4/14/2013 12:13:31 AM</DateCreated>
  <DateAccessed>4/14/2013 12:13:31 AM</DateAccessed>
  <DateModified>4/28/2013 1:40:05 AM</DateModified>
  <Path>C:\temp</Path>
  <Extension>.xml</Extension>
  <Key>ITb0cOlZQ+sGPNK3vkEhpxLMzfPCAu7pJ65vps0KOCA=</Key>
  <IV>LUax2UAZpbpgVfTFrcjYxQ==</IV>
  <GUID>83b74ec8-41f0-47c8-83b6-070ece0d6c2f</GUID>
  <Hash>SKFhPmlD6ASANDUq2DrTbA==</Hash>
</root>
Ddrak
Save a Koala, deport an Australian
Posts: 17516
Joined: Thu Jan 02, 2003 3:00 pm
Location: Straya mate!
Contact:

Re: Code Blog Topics

Post by Ddrak »

How is the metadata encrypted? Where's the key for that?

Dd
Image
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

Ddrak wrote:How is the metadata encrypted? Where's the key for that?
The meta data is also encrypted with AES, using a key derived from a user password. People can't remember keys.

So, to decrypt a file in the archive, first it needs to decrypt the meta data using the user-supplied password. Then it takes the key/IV for that file from the meta data and uses that to decrypt the file. It's probably a little more secure than using the password-derived keys for the file too. But yeah, a chain is only as strong as its weakest link, and for most products that's going to be a password-derived key.
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

The app is done. It has all the basic features like creating an archive, adding/deleting/extracting files and so forth. The blog post will have a ZIP with the complete project. This might actually be useful to mankind.
Image

The data I'm getting looks nice and encrypted. I'm not a mathematician but I believe I'm doing everything right, and again, it sure looks good and all my tests check out.

Image

Here's the main archive class:

Code: Select all

// -----------------------------------------------------------------------------
// 
// Class:       EncryptedArchive
// Author:      Mark Wing
// Date:        05/11/2013
// Purpose:     C# Encrypted Archive class for adding, deleting and 
//              extracting files.
//
// Notice:      Copyright (c) 2013 by Mark Wing.
//              You are free to use and distribute this code.
// -----------------------------------------------------------------------------


using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Xml;
using System.Security.Cryptography;
using System.Data;

namespace FileEncryptionUtilities
{
    public class EncryptedArchive
    {
        // Hard-coded salt so the same password always generates the same key.
        public byte[] Salt = { 12, 33, 82, 18, 76, 55, 90, 34, 
                                 66, 20, 99, 52, 68, 21, 87, 100, 77 };

        public byte[] Key = null;
        public int KeySize = 32; // In bytes.
        public byte[] InitVector = null;
        public int IVSize = 16;
        public string Password = "";
        public string Notes = "";
        public string RepositoryFile = "";
        public List<string> Errors = new List<string>();


        private FileStream _stream = null;

        // -----------------------------------------------------------------------------
        // Open the repository for reading.
        // -----------------------------------------------------------------------------
        public bool OpenForReading()
        {
            bool result = false;
            try
            {
                // Open the repository for reading only and share it for reading.
                _stream = new FileStream(RepositoryFile, FileMode.Open, FileAccess.Read, FileShare.Read);
                result = true;
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return result;
        }

        // -----------------------------------------------------------------------------
        // Open the repository for writing.
        // -----------------------------------------------------------------------------
        public bool OpenForWriting()
        {
            bool result = false;
            try
            {
                // Open the repository for reading only and share it for reading.
                _stream = new FileStream(RepositoryFile, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read);
                _stream.Seek(0, SeekOrigin.End); // Seek to end of file for append.
                 result = true;
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message); 
            }
            return result;
        }

        // -----------------------------------------------------------------------------
        // Close the open repository.
        // -----------------------------------------------------------------------------
        public void Close()
        {
            try
            {
                if (_stream != null)
                {
                    _stream.Close();
                    _stream.Dispose();
                    _stream = null;
                }    
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
 
            }
       }

        // -----------------------------------------------------------------------------
        // Load an input file into a buffer.
        // -----------------------------------------------------------------------------
        public byte[] LoadInputFile(string fileName)
        {
            try
            {
                // First, try to open the input file and load it into memory.
                byte[] newBuffer = null;
                FileStream inputStream = new FileStream(fileName, FileMode.Open, 
                            FileAccess.Read, FileShare.ReadWrite);
                int inputLen = (int)inputStream.Length;
                newBuffer = new byte[inputLen];
                inputStream.Read(newBuffer, 0, inputLen);
                inputStream.Close();
                inputStream.Dispose();
                return newBuffer;

            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);   
            }
            return null;
        }
 

        // -----------------------------------------------------------------------------
        // Encrypt a file and return a pile of bytes.
        // -----------------------------------------------------------------------------
        public byte[] EncryptFile(string fileName, RepositoryFile repoFile, byte[] newBuffer)
        {
            try
            {
         
                // Setup the key for this file.
                Key = Convert.FromBase64String(repoFile.Properties["Key"]);
                InitVector = Convert.FromBase64String(repoFile.Properties["IV"]);

                // Do the actual encryption.
                return EncryptBuffer(newBuffer, false);
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return newBuffer;
        }

        // -----------------------------------------------------------------------------
        // Encrypt/decrypt a buffer full of bytes.
        // -----------------------------------------------------------------------------
        public byte[] EncryptBuffer(byte[] inBuffer, bool decrypt)
        {
            try
            {
                // Use AES 256.
                Aes encryptor = Aes.Create();
                encryptor.Key = Key;
                encryptor.IV = InitVector;

                // Create the crypto stream.
                MemoryStream memStream = new MemoryStream();
                CryptoStream crypStream = null;

                if (decrypt)
                {
                    crypStream = new CryptoStream(memStream,
                                 encryptor.CreateDecryptor(),
                                 CryptoStreamMode.Write);
             
                }
                else
                {
                   crypStream = new CryptoStream(memStream,
                                    encryptor.CreateEncryptor(),
                                    CryptoStreamMode.Write);
                }

                // Encrypt/decrypt the buffer.
                crypStream.Write(inBuffer, 0, inBuffer.Length);
                crypStream.Close();
                crypStream.Dispose();

                byte[] cryptoBytes = memStream.ToArray();

                // Do some cleanup.
                memStream.Close();
                memStream.Dispose();

                return cryptoBytes;
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }

            return null;
        }

        // -----------------------------------------------------------------------------
        // Encrypt/decrypt a buffer full of bytes with a password.
        // -----------------------------------------------------------------------------
        public byte[] EncryptBufferWithPassword(byte[] inBuffer, bool decrypt)
        {
            try
            {
                DeriveKey();

                // Encrypt the buffer with the password derived key.
                return EncryptBuffer(inBuffer, decrypt);
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);                 
            }
            return null;
        }

        // -----------------------------------------------------------------------------
        // Compute the hash of a file buffer.
        // -----------------------------------------------------------------------------
        public byte[] ComputeHash(byte[] buffer)
        {
            try
            {
                MD5 calc =  MD5.Create();
                return calc.ComputeHash(buffer);                
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);            
            }
            return null;
        }      


        // -----------------------------------------------------------------------------
        // Create a new salt buffer.
        // -----------------------------------------------------------------------------
        public byte[] GimmeSomeSalt()
        {
            try
            {
                // Create a random salt.
                RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();
                byte[] saltBytes = new byte[KeySize];
                rng.GetBytes(saltBytes);
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);                
            }
            return null;
        }

        // -----------------------------------------------------------------------------
        // Derive a key from password. Use hard-coded salt so we get the same key
        // every time.
        // -----------------------------------------------------------------------------
        public void DeriveKey()
        {
            try
            {
                 // Create the password derived key.
                PasswordDeriveBytes bytes = new PasswordDeriveBytes(Password, Salt);
                Key = bytes.GetBytes(KeySize);
                InitVector = bytes.GetBytes(IVSize);
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
        }

        // -----------------------------------------------------------------------------
        // Add a file to the repository.
        // -----------------------------------------------------------------------------
        public bool AddFileToRepository(string fileToAdd)
        {
            bool result = false;
            try
            {
                // Build a file object.
                RepositoryFile info = BuildFileInfo(fileToAdd);

                if (info != null)
                {
                    // Load the file.
                    byte[] fileBuffer = LoadInputFile(fileToAdd);
                    if (fileBuffer == null)
                        return false;

                    // Compute the file's hash.
                    info.Properties.Add("Hash", Convert.ToBase64String(ComputeHash(fileBuffer)));

                    // Add any extra properties here -->

                    // Convert to document for storage in the archive.
                    XmlDocument fileDoc = info.ToXml();
   
                    // Now encrypt the XML document.
                    byte[] docBuffer = EncryptBufferWithPassword(Encoding.Unicode.GetBytes(fileDoc.InnerXml), false);       

                    // Open the archive.
                    if (OpenForWriting())
                    {
                        BinaryWriter writer = new BinaryWriter(_stream); 

                        // Write the encrypted file buffer.
                        byte[] fileEncBuffer = EncryptFile(fileToAdd, info, fileBuffer);

                        if (fileEncBuffer != null)
                        {
                            // Write the buffer sizes.
                            writer.Write(docBuffer.Length);             
                            writer.Write(fileDoc.InnerXml.Length);      
                            writer.Write(fileEncBuffer.Length);            
                            writer.Write(int.Parse(info.Properties["Size"]));
                                
                            // Write the buffers.
                            writer.Write(docBuffer);
                            writer.Write(fileEncBuffer);                           
                        }

                        // Close the archive.
                        Close();
                    }

                }
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return result;             
        }

        // -----------------------------------------------------------------------------
        // Build a file object based on a Windows file.
        // -----------------------------------------------------------------------------
        public RepositoryFile BuildFileInfo(string fileName)
        {
            RepositoryFile repoFile = null;
            try
            {
                // Make sure file exists.
                if (File.Exists(fileName))
                {
                    // Get the basic info.
                    FileInfo info = new FileInfo(fileName);

                    // Create file object and fill it.
                    repoFile = new RepositoryFile();

                    // Get all the basic file info.
                    repoFile.Properties.Add("Name", info.Name);
                    repoFile.Properties.Add("Size", info.Length.ToString());
                    repoFile.Properties.Add("DateCreated", info.CreationTime.ToString());
                    repoFile.Properties.Add("DateAccessed", info.LastAccessTime.ToString());
                    repoFile.Properties.Add("DateModified", info.LastWriteTime.ToString());
                    repoFile.Properties.Add("Extension", info.Extension.Substring(1, info.Extension.Length - 1));

                     // Generate a key for this file.
                    repoFile.GenerateKey(KeySize);      
              
                    // Generate a GUID too.
                    repoFile.GenerateGuid();

                    // Calculate folder name by stripping off drive.
                    string[] dirSplit = info.DirectoryName.Split(':');

                    if (dirSplit.Length > 1)
                    {
                        repoFile.Properties.Add("Folder", dirSplit[1]);
                    }        
                }
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return repoFile;
        }

        // -----------------------------------------------------------------------------
        // Return a list of all the files in the repository.
        // -----------------------------------------------------------------------------
        public List<RepositoryFile> ListFiles()
        {
            try
            {
                ContainerListCommand lister = new ContainerListCommand();
                return (List<RepositoryFile>) IterateThroughRepository(lister, "");
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);   
            }
            return null;
        }

        // -----------------------------------------------------------------------------
        // Load meta data for file at current file seek offset.
        // -----------------------------------------------------------------------------
        private RepositoryFile LoadFileMetaData()
        {
            try
            {
                BinaryReader reader = new BinaryReader(_stream);
                RepositoryFile newFile = new RepositoryFile();

                // Read the size values for meta and file data.
                int docEncSize = reader.ReadInt32();
                int docDecSize = reader.ReadInt32();
                int fileEncSize = reader.ReadInt32();
                int fileDecSize = reader.ReadInt32();

                // Load the meta data.
                byte[] docBuffer = reader.ReadBytes(docEncSize);
                byte[] decDocBuffer = EncryptBufferWithPassword(docBuffer, true);
                string docString = Encoding.Unicode.GetString(decDocBuffer);
                XmlDocument doc = new XmlDocument();
                doc.LoadXml(docString);

                // Build a file obect from XML document.
                newFile.FromXml(doc);

                // Add the file property if it's not there.
                if (!newFile.Properties.ContainsKey("EncryptedSize"))
                {
                    newFile.Properties.Add("EncryptedSize", fileEncSize.ToString());
                }

                return newFile;
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return null;
        }

        // -----------------------------------------------------------------------------
        // Convert file list to data table for easy binding to grid controls and such.
        // -----------------------------------------------------------------------------
        public DataTable ConvertListToTable(List<RepositoryFile> fileList)
        {
            DataTable tbl = new DataTable();
            try
            {
                tbl.Columns.Add("Name");
                tbl.Columns.Add("Folder");
                tbl.Columns.Add("Size", System.Type.GetType("System.Int32"));
                tbl.Columns.Add("Date", System.Type.GetType("System.DateTime"));
                tbl.Columns.Add("Hash");
                tbl.Columns.Add("GUID");

                foreach (RepositoryFile fileEntry in fileList)
                {
                    DataRow row = tbl.NewRow();
                    row["Name"] = fileEntry.Properties["Name"];
                    row["Folder"] = fileEntry.Properties["Folder"];
                    row["Size"] = int.Parse(fileEntry.Properties["Size"]);
                    row["Date"] = DateTime.Parse(fileEntry.Properties["DateCreated"]);
                    row["Hash"] = fileEntry.Properties["Hash"];
                    row["GUID"] = fileEntry.Properties["GUID"];
                    tbl.Rows.Add(row);
                }
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return tbl;
        }

        // -----------------------------------------------------------------------------
        // Extract a single file by ID.
        // -----------------------------------------------------------------------------
        public bool ExtractFile(string fileId, string outputFolder, bool keepArchiveFolder)
        {
            bool retVal = false;
            try
            {
                // Seek to the GUID.
                RepositoryFile repoFile = SeekToFile(fileId);

                if (repoFile != null)
                {
                    byte[] encryptedBuffer = LoadFileFromRepository(repoFile);

                    if (encryptedBuffer != null)
                    { 
                        // Now decrypt it.
                        Key = Convert.FromBase64String(repoFile.Properties["Key"]);
                        InitVector = Convert.FromBase64String(repoFile.Properties["IV"]);

                        byte[] unencryptedBuffer = EncryptBuffer(encryptedBuffer, true);

                        if (unencryptedBuffer != null)
                        { 
                            // Compute the hash of the unencrypted file.
                            string hashVal = Convert.ToBase64String(ComputeHash(unencryptedBuffer));

                            // Make sure the hashes match.
                            if (hashVal != repoFile.Properties["Hash"])
                                return false;

                            // Build the output file name.
                            string fileName = "";
                            string folder = "";

                            if (keepArchiveFolder)
                            {
                                fileName = string.Format("{0}{1}\\{2}", 
                                        outputFolder,
                                        repoFile.Properties["Folder"],
                                        repoFile.Properties["Name"]);
                                folder = string.Format("{0}{1}",
                                        outputFolder,
                                        repoFile.Properties["Folder"]);
                            }
                            else
                            {
                                 
                                fileName = string.Format("{0}\\{1}", 
                                        outputFolder,
                                        repoFile.Properties["Name"]);
                                folder = outputFolder;
                      
                            }
                            
                            // Create the output folder.
                            try
                            {
                                Directory.CreateDirectory(folder);
                            }
                            catch (Exception)
                            {
                              
                            }

                            // Write the unencrypted file.
                            FileStream  outStream = new FileStream(fileName, FileMode.Create, FileAccess.Write);
                            BinaryWriter writer = new BinaryWriter(outStream);
                            writer.Write(unencryptedBuffer);
                            outStream.Close();
                            writer.Dispose();
                            outStream.Dispose();
                        }

                    }
                }
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return retVal;
        }

        // -----------------------------------------------------------------------------
        // Load an encrypted file from the repository if we already have its info.
        // -----------------------------------------------------------------------------
        private byte[] LoadFileFromRepository(RepositoryFile repoFile)
        {
            try
            {
                if (OpenForReading())
                {
                    _stream.Seek(long.Parse(repoFile.Properties["FileOffset"]), SeekOrigin.Begin);
                    BinaryReader reader = new BinaryReader(_stream);
                    byte[] buffer = reader.ReadBytes(int.Parse(repoFile.Properties["EncryptedSize"]));
                    reader.Dispose();
                    Close();

                    return buffer;
                }
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }

            return null;
        }

        // -----------------------------------------------------------------------------
        // Allow for bulk processing on the repository.
        // -----------------------------------------------------------------------------
        public Object IterateThroughRepository(IFileContainerCommand command, string param1)
        {
            try
            {     
                if (OpenForReading())
                {
                    command.Param1(param1);

                    if (command.Begin(ref _stream))
                    {
                        RepositoryFile fileEntry = null;

                        // Keep reading files until we run out of space.
                        do
                        {
                            // Load this file's meta data.
                            fileEntry = LoadFileMetaData();

                            if (fileEntry != null)
                            {
                                // Jot down the offset of this file.
                                if (fileEntry.Properties.ContainsKey("FileOffset"))
                                {
                                    fileEntry.Properties.Remove("FileOffset");
                                }
                                fileEntry.Properties.Add("FileOffset", _stream.Position.ToString());


                                if (!command.ProcessFile(fileEntry))
                                    break;

                                // Seek past file data.
                                _stream.Seek(long.Parse(fileEntry.Properties["EncryptedSize"]), 
                                                SeekOrigin.Current);

                            }

                        } while (fileEntry != null);

                        // Close the file.
                        Close();

                        return command.End();
                    }
                }
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return null;
        }

        // -----------------------------------------------------------------------------
        // Seek to file by its unique ID.
        // -----------------------------------------------------------------------------
        public RepositoryFile SeekToFile(string fileId)
        {
            try
            {
                ContainerSeekToFile seeker = new ContainerSeekToFile();
                return (RepositoryFile)IterateThroughRepository(seeker, fileId);
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }

            return null;
        }

        // -----------------------------------------------------------------------------
        // Seek to file by its unique ID.
        // -----------------------------------------------------------------------------
        public void ExtractMetaData(string outputXmlFile)
        {
            try
            {
                ExtractAllMetaData extractor = new ExtractAllMetaData();
                XmlDocument doc = (XmlDocument)IterateThroughRepository(extractor, "");
                doc.Save(outputXmlFile);
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }

        }

        // -----------------------------------------------------------------------------
        // Delete files from archive given a list of GUIDs.
        // -----------------------------------------------------------------------------
        public bool DeleteFilesFromRepository(List<string> fileIds)
        {
            bool retVal = false;
            try
            {
                List<RepositoryFile> allFiles = ListFiles();

                if (allFiles != null && fileIds.Count > 0)
                {
                    // Create and open temp archive.
                    string tempFileName = RepositoryFile + ".copy_of";
                    FileStream tempArchive = new FileStream(tempFileName, 
                                        FileMode.Create, FileAccess.Write, FileShare.Read);
                    BinaryWriter writer = new BinaryWriter(tempArchive);

                    // Open the main archive.
                    if (OpenForReading())
                    {
                        BinaryReader reader = new BinaryReader(_stream);

                        // Loop through all files in archive.
                        foreach (RepositoryFile repoFile in allFiles)
                        {
                            // See if it's not in delete list.
                            if (!fileIds.Contains(repoFile.Properties["GUID"]))
                            {
                                // Re-encrypt the meta data.
                                XmlDocument fileDoc = repoFile.ToXml();
                                byte[] docBuffer = EncryptBufferWithPassword(Encoding.Unicode.GetBytes(fileDoc.InnerXml), false);      

                                // Copy it to the new archive.
                                writer.Write(docBuffer.Length);
                                writer.Write(fileDoc.InnerXml.Length);
                                writer.Write(int.Parse(repoFile.Properties["EncryptedSize"]));
                                writer.Write(int.Parse(repoFile.Properties["Size"]));

                                // Load the encrypted file as-is.
                                _stream.Seek(long.Parse(repoFile.Properties["FileOffset"]), SeekOrigin.Begin);
                                 byte[] fileEncBuffer = reader.ReadBytes(int.Parse(repoFile.Properties["EncryptedSize"]));
          
                                // Write the buffers.
                                writer.Write(docBuffer);
                                writer.Write(fileEncBuffer);
                            }
                        }
                        reader.Dispose();
                    }
                    
                    // Close both archives.
                    Close();
                    tempArchive.Close();

                    // Now swap the files out.
                    File.Delete(RepositoryFile);
                    File.Move(tempFileName, RepositoryFile);

                    retVal = true;
                }
            }
            catch (Exception ex)
            {
                Errors.Add(ex.Message);
            }
            return retVal;
        }

    }

  

    // -----------------------------------------------------------------------------
    // Repository file class.
    // -----------------------------------------------------------------------------
    public class RepositoryFile
    {
        // Name/value pair list of file properties.
        public Dictionary<string, string> Properties = new Dictionary<string, string>();

        // -----------------------------------------------------------------------------
        // Populate from an XML document.
        // -----------------------------------------------------------------------------
        public void FromXml(XmlDocument doc)
        {
            try
            {
                foreach (XmlNode topNode in doc.DocumentElement.ChildNodes)
                {
                    Properties.Add(topNode.Name, topNode.InnerText);
                }
            }
            catch (Exception)
            {
             
            }
        }        
        
        // -----------------------------------------------------------------------------
        // Convert the file properties to XML document.
        // -----------------------------------------------------------------------------
        public XmlDocument ToXml()
        {
            XmlDocument doc = null;
            try
            {
                doc = new XmlDocument();
                doc.LoadXml("<root></root>");
                foreach (KeyValuePair<string, string> fileProp in Properties)
                {
                    // Append a new node in the form of <property>value</property>
                    XmlNode node = doc.CreateElement(fileProp.Key);
                    node.InnerText = fileProp.Value;
                    doc.DocumentElement.AppendChild(node);
                }
            }
            catch (Exception)
            {
             
            }

            return doc;
        }

        // -----------------------------------------------------------------------------
        // Generate a separate key and IV for every file in the repository.
        // -----------------------------------------------------------------------------
        public void GenerateKey(int keySizeBytes)
        {
            try
            {
                Aes encryptor = Aes.Create();
                encryptor.KeySize = keySizeBytes * 8;
                encryptor.GenerateKey();
                encryptor.GenerateIV();
                Properties["Key"] = Convert.ToBase64String(encryptor.Key);
                Properties["IV"] = Convert.ToBase64String(encryptor.IV);  
            }
            catch (Exception)
            {
             
            }
        }

        // -----------------------------------------------------------------------------
        // Generate a GUID for this file, since we're allowing duplicate names.
        // -----------------------------------------------------------------------------
        public void GenerateGuid()
        {
            try
            {
                Properties["GUID"] = Guid.NewGuid().ToString();
            }
            catch (Exception)
            {
            
            }
        }
        
    }
}
Here's the batch helper class:

Code: Select all

// -----------------------------------------------------------------------------
// 
// Class:       EncryptedArchiveBatch
// Author:      Mark Wing
// Date:        05/11/2013
// Purpose:     C# Batch processing for encrypted file archive class.
//
// Notice:      Copyright (c) 2013 by Mark Wing.
//              You are free to use and distribute this code.
// -----------------------------------------------------------------------------

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Xml;

namespace FileEncryptionUtilities
{

    // -----------------------------------------------------------------------------
    // Generic interface for processing file batch operations.
    // -----------------------------------------------------------------------------

    public interface IFileContainerCommand
    {
        bool Begin(ref FileStream repoStream);
        object End();
        bool ProcessFile(RepositoryFile repoFile);
        void Param1(string param);
    }


    // -----------------------------------------------------------------------------
    // Interface implementation to extract all meta data.
    // -----------------------------------------------------------------------------

    class ExtractAllMetaData : IFileContainerCommand
    {
        private XmlDocument _doc = new XmlDocument();
 
        bool IFileContainerCommand.Begin(ref FileStream repoStream)
        {
            _doc.LoadXml("<root></root>");
            return true;
        }

        bool IFileContainerCommand.ProcessFile(RepositoryFile fileEntry)
        {
            bool retVal = true;
            try
            {
                // Add a file element to output doc.
                XmlElement fileNode = _doc.CreateElement("File");
                _doc.DocumentElement.AppendChild(fileNode);
                
                // Tack on tags for this file.
                foreach (KeyValuePair<string, string> fileProp in fileEntry.Properties)
                {
                    // Append a new node in the form of <property>value</property>
                    XmlNode node = _doc.CreateElement(fileProp.Key);
                    node.InnerText = fileProp.Value;
                    fileNode.AppendChild(node);
                }
    
            }
            catch (Exception)
            {
            }
            return retVal;
        }

        void IFileContainerCommand.Param1(string param)
        {
        }

        object IFileContainerCommand.End()
        {
            return _doc;
        }

    }
    // -----------------------------------------------------------------------------
    // Interface implementation to seek to a specfic file in the archive.
    // -----------------------------------------------------------------------------

    class ContainerSeekToFile : IFileContainerCommand
    {
        private RepositoryFile _repoFile = null;
        private string fileId = "";

        bool IFileContainerCommand.Begin(ref FileStream repoStream)
        {
            _repoFile = null;
            return true;
        }

        bool IFileContainerCommand.ProcessFile(RepositoryFile fileEntry)
        {
            if (fileEntry.Properties["GUID"] == fileId)
            {
                _repoFile = fileEntry;
                return false;
            }
            else
            {
                return true;
            }
        }

        void IFileContainerCommand.Param1(string param)
        {
            fileId = param;
        }

        object IFileContainerCommand.End()
        {
            return _repoFile;
        }
      
    }
 
    // -----------------------------------------------------------------------------
    // File listing interface.
    // -----------------------------------------------------------------------------

    class ContainerListCommand :  IFileContainerCommand
    {
        private List<RepositoryFile> _fileList = null;
        FileStream _stream = null;

        bool IFileContainerCommand.Begin(ref FileStream repoStream)
        {
            _stream = repoStream;
            _fileList = new List<RepositoryFile>();
            return true;
        }

        void IFileContainerCommand.Param1(string param)
        { }

        object IFileContainerCommand.End()
        {
            return _fileList;
        }

        bool IFileContainerCommand.ProcessFile(RepositoryFile fileEntry)
        {
            try
            {
                // Add to output list.
                _fileList.Add(fileEntry);

                 // Tell processor to keep going.
                return true;
            }
            catch (Exception)
            {

            }
            return false;
        }
    }
}
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

The post is published! This should be an actually useful utility.

C# Sample App: Encrypted File Archive Utility
User avatar
Taxious
Rum Guzzler
Posts: 5056
Joined: Fri Apr 18, 2003 10:16 am
Location: Denver, CO

Re: Code Blog Topics

Post by Taxious »

Cool stuff, good read!
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Freecare Spiritwise
Grand Pontificator
Posts: 3015
Joined: Thu Mar 13, 2003 5:35 pm

Re: Code Blog Topics

Post by Freecare Spiritwise »

Taxious wrote:Cool stuff, good read!
Thanks, Tax!
Post Reply