Big Data is a new class of technology that gives businesses more insight into their massive data sets, allowing them to make better business decisions and satisfy customers. Big data systems are also a desirable target for hackers due to the aggregation of their data.The Hadoop Distributed File System (HDFS) stores massive data in the Hadoop framework. Since HDFS does not safeguard data privacy, encrypting the file is the right way to protect the stored data in HDFS but takes a long time. In this paper, regarding privacy concerns, we use different compression-type data storage file formats with the proposed hybrid user-defined function (HUDF)based on an XOR-Onetime pad with AES to securedata in HDFS. In this way, we provide a dual level of encryptionby maskingselective data and whole data in the file. Our experiment demonstrates that this scheme offersoverall data security and also faster processing time than the conventional methods. The proposed HUDF with ORC, Zlib (Z) file format (HUDF-ORC-Z) gives 9-10% better performanceresults than 2DES and other method. Finally, we efficiently utilized the space, improved query processing time,and decreased data load timewith the Hive engine.